ai bots

pay-up-or-stop-scraping:-cloudflare-program-charges-bots-for-each-crawl

Pay up or stop scraping: Cloudflare program charges bots for each crawl

“Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho—and then giving that agent a budget to spend to acquire the best and most relevant content,” Cloudflare said, promising that “we enable a future where intelligent agents can programmatically negotiate access to digital resources.”

AI crawlers now blocked by default

Cloudflare’s announcement comes after rolling out a feature last September, allowing website owners to block AI crawlers in a single click. According to Cloudflare, over 1 million customers chose to block AI crawlers, signaling that people want more control over their content at a time when Cloudflare observed that writing instructions for AI crawlers in robots.txt files was widely “underutilized.”

To protect more customers moving forward, any new customers (including anyone on a free plan) who sign up for Cloudflare services will have their domains, by default, set to block all known AI crawlers.

This marks Cloudflare’s transition away from the dreaded opt-out models of AI scraping to a permission-based model, which a Cloudflare spokesperson told Ars is expected to “fundamentally change how AI companies access web content going forward.”

In a world where some website owners have grown sick and tired of attempting and failing to block AI scraping through robots.txt—including some trapping AI crawlers in tarpits to punish them for ignoring robots.txt—Cloudflare’s feature allows users to choose granular settings to prevent blocks on AI bots from impacting bots that drive search engine traffic. That’s critical for small content creators who want their sites to still be discoverable but not digested by AI bots.

“AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source—depriving content creators of revenue, and the satisfaction of knowing someone is reading their content,” Cloudflare’s blog said. “If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”

Disclosure: Condé Nast, which owns Ars Technica, is a partner involved in Cloudflare’s beta test.

This story was corrected on July 1 to remove publishers incorrectly listed as participating in Cloudflare’s pay-per-crawl beta.

Pay up or stop scraping: Cloudflare program charges bots for each crawl Read More »

so-much-for-free-speech-on-x;-musk-confirms-new-users-must-soon-pay-to-post

So much for free speech on X; Musk confirms new users must soon pay to post

100 pennies for your thoughts? —

The fee, likely $1, is aimed at stopping “relentless” bots, Musk said.

So much for free speech on X; Musk confirms new users must soon pay to post

Elon Musk confirmed Monday that X (formerly Twitter) plans to start charging new users to post on the platform, TechCrunch reported.

“Unfortunately, a small fee for new user write access is the only way to curb the relentless onslaught of bots,” Musk wrote on X.

In October, X confirmed that it was testing whether users would pay a small annual fee to access the platform by suddenly charging new users in New Zealand and the Philippines $1. Paying the fee enabled new users in those countries to post, reply, like, and bookmark X posts.

That test was deemed the “Not-A-Bot” program, and it’s unclear how successful it was at stopping bots. But X deciding to expand the program seems to suggest that the test must have had some success.

Musk has not yet clarified when X’s “small fee” might be required for new users, only confirming in a later post that any new users who avoid paying the fee will be able to post after three months. Ars created new accounts on the web and in the app, and neither signup required any fees yet.

Although Musk’s posts only mention paying for “write access,” it seems likely that the other features limited by the “Not-A-Bot” program will also be limited during those three months for any users who do not pay the fee, too. An X account called @x_alerts_ noticed on Sunday that X was updating its web app text that was seemingly enabling the “Not-A-Bot” program.

“Changes have been detected in the texts of the X web app!” @x_alerts_ wrote, noting that the altered text seemed to limit not just posting and replying, but also liking and bookmarking X posts.

“It looks like this text has been in the app, but they recently changed it, so not sure whether it’s an indication of launch or not!” the user wrote.

Back when X launched the “Not-A-Bot” program, Musk claimed that charging a $1 annual fee would make it “1000X harder to manipulate the platform.” In a help center post, X said that the “test was developed to bolster our already significant efforts to reduce spam, manipulation of our platform, and bot activity.”

Earlier this month, X warned users it was widely purging spam accounts, TechCrunch noted. X Support confirmed that follower counts would likely be impacted during that purge, because “we’re casting a wide net to ensure X remains secure and free of bots.”

But that attempt to purge bots apparently did not work as well as X hoped. This week, Musk confirmed that X is still struggling with “AI (and troll farms)” that he said are easily able to pass X’s “are you a bot” tests.

It’s hard to keep up with X’s inconsistent messaging on its bot problem since Musk took over. Last summer, Musk told attendees of The Wall Street Journal’s CEO Council that the platform had “eliminated at least 90 percent of scams,” claiming there had been a “dramatic improvement” in the platform’s ability to “detect and remove troll armies.”

At that time, experts told The Journal that solving X’s bot problem was nearly impossible because spammers’ tactics were always evolving and bots had begun using generative AI to avoid detection.

Musk’s plan to charge a fee to overcome bots won’t work, experts told WSJ, because anyone determined to spam X can just find credit cards and buy disposable phones on the dark web. And any bad actor who can’t find what they need on the dark web could theoretically just wait three months to launch scams or spread harmful content like disinformation or propaganda. This leads some critics to wonder what the point of charging the small fee really is.

When the “Not-A-Bot” program launched, X Support directly disputed critics’ claims that the program was simply testing whether charging small fees might expand X’s revenue to help Musk get the platform out of debt.

“This new test was developed to bolster our already successful efforts to reduce spam, manipulation of our platform, and bot activity, while balancing platform accessibility with the small fee amount,” X Support wrote on X. “It is not a profit driver.”

It seems likely that Musk is simply trying everything he can think of to reduce bots on the platform, even though it’s widely known that charging a subscription fee has failed to stop bots from overrunning other online platforms (just ask frustrated fans of World of Warcraft). Musk, who famously overpaid for Twitter and has been climbing out of debt since, has claimed since before the Twitter deal closed that his goal was to eliminate bots on the platform.

“We will defeat the spam bots or die trying!” Musk tweeted back in 2022, when a tweet was still a tweet and everyone could depend on accessing Twitter for free.

So much for free speech on X; Musk confirms new users must soon pay to post Read More »