Perplexity Defends Its AI Assistants Against Cloudflare’s Blocking Claims

Perplexity is fighting back against Cloudflare’s recent decision on their AI Assistants, arguing that they shouldn’t be classified as traditional web crawlers. 

Perplexity defends Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. Perplexity claims that their technology is completely different with its AI Assistants, as Perplexity’s AI Assistants only fetch content in response to queries from users instead of systematically crawling and indexing websites.

Perplexity Argument

Perplexity argument that its system is not the typical web crawler. Contrary to the traditional robots that search the web the web, its AI assistants only fetch content when they are triggered by specific requests from users. 

For example when a user requests for reviews of recent restaurants and the AI retrieves and summarizes relevant content upon demand instead of collecting and storing massive quantities of information.

In this regard, Perplexity compares its service with Google’s on-demand fetching service for specific tasks, like the reading of text and checking the authenticity of content on websites. 

But, Perplexity distinguishes its AI Assistants by insisting that they only operate in the context of individual user actions, not actively searching and indexing web content. According to the company that their actions are different from traditional web crawlers.

The AI Assistant’s Role

Perplexity then clarifies that its assistants don’t retain or use the information they collect to build models. This is different from traditional crawlers that constantly search for huge number of information without considering for the intent of the user. 

Perplexity claims that their AI is purely an extension of the request of the user that is designed to give timely, relevant answers without compromising the privacy of users or participating in wide web scraping.

Criticism of Cloudflare’s Approach

In addition to defending of the company’s AI agents, Perplexity also criticized Cloudflare’s infrastructure for misclassifying legitimate user-generated traffic as scraping that is malicious. 

Perplexity argued that the inability of Cloudflare to distinguish between AI bots that are harmful and assistants causes excessive blocking, which hurts companies that are not engaged in illegal scraping.

Perplexity said the fact that Cloudflare’s actions are based on a misunderstanding of how the technology works and the business believes that AI Assistants shouldn’t be subject to the same limitations as the traditional web crawlers.

What This Means for AI, Web Crawling and SEOs?

Perplexity’s response illustrates the growing difficulty in separating legitimate AI-powered tools from bots with a traditional approach that may be engaging in more destructive or damaging behavior. 

In the future, as AI plays a greater role in interactions on the web, the issue of AI-driven web traffic will increase. The debate raises significant questions regarding the way we define and control the automated traffic on the web, particularly when AI technology is integrated into our daily tasks.

For SEO experts, the issue highlights the complexity of managing bots and traffic filtering. Cloudflare’s strategy is to shield websites from scraping that is harmful but it’s evident that AI tools such as Perplexity push the boundaries of what is considered legitimate bot activities. 

Read Perplexity’s response here: Agents or Bots? Making Sense of AI on the Open Web