Balancing security and privacy in the battle against AI scraping

In recent years, as adoption of generative AI has skyrocketed, the dynamics of the internet have changed. Automated access has long been a feature of the web: search engines crawl pages to make information discoverable, researchers use bots to collect data, and businesses use them to monitor markets. But the rapid expansion of generative AI introduced a new dimension to this long-standing relationship between bots and the web. Training and improving models required access to massive quantities of online data, and automated collection grew at an unprecedented scale.

For many website operators, this surge in bot traffic meant increased strain on their infrastructure. Websites built with human browsing behaviour in mind suddenly had to accommodate large volumes of automated requests, which led to higher operational costs and, in some cases, decreased performance and availability. 

Yet automation itself is not the problem; bots remain essential to internet use. The problem is scale; scraping has become a for-profit industry and a major economic, security, and policy challenge. The key question is not whether automated access should exist, but how to distinguish beneficial uses from harmful or excessive extraction and ensure that protective measures do not create unnecessary barriers for legitimate users.

The cybersecurity industry explicitly recognises web scraping as an automated threat. In response to this scourge of scrapers, websites have begun incorporating anti-bot systems, including browser and device fingerprinting, behavioural analysis, CAPTCHAs, and network-level detection techniques. 

However, despite its operational impact, scraping occupies a persistent legal and regulatory grey zone. The act of collecting publicly accessible data is, in most jurisdictions, not illegal. In Europe, for example, there is no general prohibition on collecting this type of data, but ‘publicly available’ does not necessarily mean ‘freely reusable’. Instead, legality depends on factors such as the scale of extraction, the intent behind the activity, and how the collected data is used.

This ambiguity lies at the heart of today’s debate. Web scraping should not be regulated as a binary legal act, but rather governed through a risk-based framework that takes context into account.

At the same time, any meaningful discussion of regulation should also consider the potential implications of defensive measures deployed by websites, as some anti-bot techniques may raise questions around proportionality, transparency, and their impact on user privacy.

Scraping as a business and security problem

Web scraping is the automated, periodic, or continuous retrieval of data from web pages for reuse elsewhere. In practice, scraping is rarely performed by a single script; it is typically conducted through distributed networks of automated agents (bots) designed to collect content at scale while mimicking legitimate user behavior.

Some uses of scraping are arguably societally useful, such as search engine indexing, archiving, or academic research. Others are commercial: large-scale data aggregation, price intelligence, content republishing, or competitive monitoring. In more problematic cases, scraped data is resold, used to manipulate markets, or combined with other datasets to infer sensitive attributes about individuals.

E-commerce platforms are frequent targets. These websites expose high-value information (prices, availability, promotions) while assuming only a small fraction of page views convert into purchases. Every request carries a cost: server infrastructure, backend computation, data retrieval, and rendering. Scraping disrupts this balance by generating massive volumes of non-converting traffic. 

This automated abuse has significant impacts on revenue. Direct losses stem from increased operational costs without corresponding income. Indirect losses arise when scraping traffic distorts analytics, undermines demand forecasting, or degrades legitimate user experiences through congestion and latency. At extreme scales, denial-of-service conditions render services unavailable.

From the website operator’s perspective, scraping is not an abstract legal issue but a concrete operational threat affecting availability, integrity, and economic viability. However, the legal classification of scraping remains complex. For website operators, this often makes responding through legal channels uncertain and significantly slower than the pace at which automated scraping occurs, making technical defences the primary means of protection.

Terms of service further complicate this landscape. Cases such as LinkedIn vs hiQ, where LinkedIn attempted to restrict a third party from scraping publicly available profiles, illustrate how the boundaries between legitimate access and prohibited extraction remain context- and jurisdiction-dependent. This legal uncertainty has pushed many websites to rely on technical measures, including anti-bot systems, to protect their infrastructure.

Defending websites: the rise of anti-bot technologies

Websites use increasingly sophisticated anti-bot systems to protect against automated abuse. In response to the evolving techniques adopted by attackers, both commercial actors and academic researchers have developed a wide range of detection and mitigation approaches. 

Modern defences typically rely on probabilistic inference, correlating multiple signals across requests and sessions to estimate the likelihood that an interaction originates from a human user or an automated agent. From a security and operational standpoint, the primary objective is to accurately identify abusive traffic while minimising disruption to legitimate users. However, as these techniques become more advanced and pervasive, it is increasingly important to consider their broader implications from a policy and user perspective.

The privacy cost of fighting bots

Recent research has begun to examine anti-bot technologies through the lens of user privacy and transparency. Modern bot mitigation strategies may involve extensive data collection, including device fingerprinting, behavioural signals, and cross-session identifiers. 

While these techniques are often justified by security requirements, they can operate with limited visibility to end users and enable forms of persistent identification. For example, browser characteristics, installed fonts, screen resolution, interaction patterns, and network information can be combined to create a unique profile of a device, even when a user restricts traditional tracking mechanisms used for targeted advertisement, such as cookies.

In seeking to protect services from automated abuse, websites may deploy mechanisms that rely on continuous monitoring and behavioural inference. For instance, detecting sophisticated bots may require analysing how quickly users interact with pages, how they move through a website, or whether their browsing patterns resemble automated activity. 

While useful for distinguishing bots from legitimate users, these approaches raise questions about proportionality and transparency: what data is collected, for how long it is retained, and whether users understand how they are being evaluated. Furthermore, legitimate users can occasionally experience increased friction such as CAPTCHAs, blocking, or seeing different content when the system misclassifies them. This is particularly the case when using tools or configurations for increased privacy.

Rather than framing this dynamic as an inherent flaw of anti-bot systems, it is more accurately understood as a design trade-off. The challenge for both industry and policymakers lies in balancing effective protection against abuse with transparency, fairness, and respect for user privacy, ensuring that defensive measures remain proportionate to the risks they aim to address.

Towards risk-based governance

Calls to either prohibit scraping entirely or declare all public data freely reusable fail to capture the technical and economic realities of the web. Scraping covers a spectrum of behaviours; occasional requests for research or interoperability differ fundamentally from industrial-scale extraction designed to generate economic value. 

Binary regulation ignores key dimensions: scale (occasional access versus mass automation), intent (research, competition, and exploitation), and downstream use (analysis, resale, profiling, and manipulation). Web scraping is neither inherently lawful nor inherently abusive. Its impact depends on how, why, and at what scale it is deployed, as does that of the technologies designed to stop it. 

A more sustainable approach is risk-based governance focusing on harm thresholds linked to scale and automation, accountability for safeguards circumvention, and obligations tied to downstream data use. This is increasingly important as the economic and societal impact of large-scale scraping grows, fuelling the training of machine learning and generative AI systems. 

Websites are entitled to defend their infrastructure and business models, but anti-bot technologies should remain proportionate, transparent, and aligned with data protection principles. Overly intrusive defences shift the cost of mitigation onto users and risk normalising opaque practices.

Reconciling security-by-design with privacy-by-design is necessary. Effective policy must move beyond simplistic legal categories toward nuanced frameworks that balance innovation, competition, security, and fundamental rights.