Perplexity wants to change the way we use the internet, but AI search startup backed by Jeff Bezos The company is violating its own rules in doing so. According to a report, the company appears to be ignoring a widely accepted web standard, the Robots Exclusion Protocol, to scrape parts of the web that operators do not want to be accessed by bots. Developer Rob Knight what was this week Confirmed by Wired,
Perplexity's service summarizes articles on the web, and claims it provides “reliable answers” “without the need to click on different links,” as noted in a report. blog postTo do this, Wired and Knight found that Perplexity intentionally ignores code (robots.txt files) written to block web crawlers. Both reports found that Perplexity uses an unlisted IP address to transmit these robots.txt files and scrape websites. Wired claims that its website blocked Perplexity's web crawler in early 2024, but the AI search engine is still able to summarize its articles in detail.
Despite this, Perplexity claims to respect the Robot Exclusion Protocol Documentation on its website. Perplexity CEO Arvind Srinivas told Wired that the reporters had “deep and fundamental misunderstandings about Perplexity and how the internet works,” but he did not directly dispute the findings. Gizmodo has contacted Perplexity for a more detailed response and will update the article if we receive a reply.
In addition, Perplexity is currently facing legal threats for breaking some other internet rules: copyright infringement. Forbes reportedly said Legal action threatened against Perplexity This week, after accusing an AI startup of Mimicking Forbes' reporting Without proper credit. Original reporting by Forbes Former Google CEO Eric Schmidt's AI drone ventureAnd Perplexity created AI-generated articles, podcasts and videos using Forbes' text and images. Forbes' executive editor called Perplexity on X at the beginning of the month.
Perplexity's product, although useful, redirects traffic to the Internet. Google also indexes webpages and provides brief AI summaries, but it directs traffic directly to the web pages where the information comes from. Perplexity is effectively writing detailed AI articles so users won't click on websites, which breaks the business model of digital media.
OpenAI has forged a Partnerships with media companies To solve this problem, they must pay upfront to license the content, and Perplexity Reportedly working on similar content partnershipsBut instead of paying a fixed fee for content like OpenAI, Perplexity aims to share revenue. But these partnerships don’t exist yet, so for now, Perplexity is getting past the paywall and scraping websites to grab all the information it needs to power its AI answers.