Reducing the Impact of Crawlers on Kernel.org: Technical Solutions

Kernel.org, the official repository of the Linux kernel, is facing growing pressure on its servers due to automated crawlers. These tools, used to collect data from websites, consume a significant portion of server resources, particularly computing power.

Kernel.org handles about 6 million daily requests for individual commit pages, with roughly a third of these managing to bypass protection systems. This behavior has a significant impact on the infrastructure, with 14-16 CPU cores out of 90 constantly busy converting Git commits to HTML for crawlers.

The main problem is that crawlers access commit data one by one instead of cloning repositories and working locally. This inefficient approach increases the server load, despite the data being freely accessible via Git.

The architecture of git.kernel.org, with over 1.48 million commits and hundreds of forks, offers an enormous number of URLs that crawlers can access, further worsening the situation.

Kernel.org has attempted various solutions, such as blocking specific IPs and networks, but crawlers have adapted by using residential and mobile proxy networks. Recently, Anubis, a proof-of-work system, was implemented, requiring users to solve a challenge before accessing the site. However, crawlers have also developed methods to overcome this barrier.

In response to these challenges, Kernel.org is considering more drastic measures, such as reducing the number of URLs accessible to crawlers and limiting the most resource-intensive actions for anonymous users. Despite these restrictions, the Linux kernel repositories and development data will remain publicly accessible.

Prerequisites

• Access to a system with an Internet connection
• Basic knowledge of Git and Git repositories
• Ability to navigate git.kernel.org
• Knowledge of basic web crawling and scraping concepts
• Access to Konstantin Ryabitsev's posts on Kernel.org for further details

Procedure: How to Reduce the Impact of Crawlers on Kernel.org

Following this guide, you will learn how to configure Anubis to mitigate the impact of crawlers on Kernel.org.

• Access the server hosting Kernel.org.
• Install Anubis following the official instructions. You can find the Anubis Git repository at this address: https://github.com/kernelci/anubis.
• Configure Anubis to block crawlers. Modify the Anubis configuration file to set the difficulty level of the proof-of-work. For example, you can set the difficulty parameter to a high value to make it harder for crawlers to solve the challenge.
• Start Anubis and verify that it is running correctly. You can do this by checking the Anubis logs to ensure there are no errors.
• Monitor the impact of Anubis on crawlers. Check the Kernel.org logs to see if the number of crawler requests has decreased.
• If necessary, adjust the Anubis configuration to further optimize performance. For example, you can modify the block_time parameter to block crawlers for a longer period.

Verification and Troubleshooting

At the end of this guide, you will be able to test the system's functionality and resolve any issues related to crawlers on Kernel.org.

Functionality Test

• CPU Load Verification: Monitor CPU usage on Kernel.org servers. If about 20% of the total computing power is dedicated to handling automated scrapers, the system is under pressure.
• Request Check: Verify the number of daily requests for individual commit pages. If they exceed 6 million, it is likely that crawlers are overloading the system.
• Traffic Analysis: Use traffic monitoring tools to identify the sources of requests. If a significant portion comes from suspicious IPs or proxy networks, action is needed.

Troubleshooting

• Blocking IPs and Networks: Implement measures to block suspicious IPs and networks. However, keep in mind that crawlers can adapt by using residential and mobile proxies.
• Implementing Anubis: Configure Anubis to require a proof-of-work from visitors. This can reduce the number of requests, but crawlers might adapt by solving more complex challenges.
• Limiting Crawlable URLs: Consider reducing the number of URLs accessible to crawlers and limiting actions that consume many resources for anonymous users.
• Maintaining Public Access: Ensure that the Linux kernel repositories and development data remain publicly accessible, despite traffic control measures.

For further details, refer to Ryabitsev's post Creepy Crawlies.

Educational Summary and Practical Invitation

org, using about 20% of the total CPU power of the project. This problem is mainly caused by the request for individual commit pages, despite the data being freely accessible via Git.

• Current Impact: Kernel.org handles 6 million requests per day, with 14-16 CPU cores always busy converting Git commits to HTML for crawlers.
• Solutions Implemented: Kernel.org has implemented Anubis, a proof-of-work system, and attempted to block certain IPs and networks, but crawlers have adapted their strategies.

To further explore the problem and proposed solutions, refer to the post by Konstantin Ryabitsev.

Test yourself: analyze the traffic of your website and identify any bots or crawlers that might compromise performance. Implement solutions similar to those described to optimize your server's resources.