The Cloudflare Blog: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 3, 2025)
Internet utility, Cloudflare, accuses Perplexity of obscuring its browser user agent (the way browsers describe themselves to web servers) in order to skirt firewall and robot rules. CF penalized Perplexity by removing it from the list of verified bots.
We received complaints from customers who had both disallowed Perplexity crawling activity in their robots.txt files and also created WAF rules to specifically block both of Perplexity’s declared crawlers: PerplexityBot and Perplexity-User.
Cloudflare then ran tests with new, secret websites to confirm this sneaky behavior.
To Perplexity’s credit, I don’t think many people using the web would expect to be blocked from visiting a website, so perhaps there is some gray area here. Is a Perplexity truly a robot or is it fundamentally controlled by a human?
I don’t like that Perplexity is being sneaky, but I also think these new AI tools push the envelope of how the web is glued together. Technology and standards will have to evolve quickly.
Leave a Reply