Pay, AI BOT: This is the message from a major company on how the Internet works

Artificial intelligence companies may find it difficult to reach the entire web to train large language models after the internet infrastructure provider said this week You will prevent artificial intelligence data crawling By default.

It is the latest front that opens into a continuous battle between content and artificial intelligence developers who use this content for training Temporary artificial intelligence models. In court, the authors and the content of the content sue the main artificial intelligence companies for compensation, saying that the copyright content has been used without permission. (Disclosure: Zif Davis, the parent company CNET, filed a lawsuit against Openai, claimed that it had violated the copyright of ZifF Davis in training and operating artificial intelligence systems.)

While content service providers are looking for compensation for the information used to train models in the past, the Cloudflare movement represents a new defensive measure in exchange for future efforts to train models.

But it is not only a matter of banning reptiles: Cloudflare says he wants it Create a market Where artificial intelligence companies can pay to crawl to a site and a healing, which means that the provider of this information is paid, and the developer of artificial intelligence gets permission.

“This content is the fuel that operates artificial intelligence engines, and therefore it is fair that the content creators will be compensated directly for that,” said Matthew Prinos, CEO of Cloudflare in A. Blog post.

Why do web sites want to prevent artificial intelligence crawling

The crawls – robots that visit and copy the information from the website – are a vital component of the connected internet. This is how search engines such as Google are known to what is on different websites, and how the latest information can serve you from places like CNET.

AI Crawles introduces distinctive challenges for web sites. For anyone, it can be aggressive and generation Unemployed levels of traffic For smaller sites. It also provides a little bonuses on the scraping: If Google crawls a site for the search engine results, it is possible that the traffic to this site is likely to include it in the search results. It may not mean crawling for training data any additional or even less traffic movement, if people stop visiting the site and rely on the artificial intelligence model.

Read more: Artificial Intelligence Basics: 29 ways to make Gen AI work for you, according to our experts

For this reason, executive officials from major web sites such as Pinterest, Reddit and many major publishing companies (including ZifFa Davis, which owns CNET) to “Cloudflare’s BewR” in data.

“The full ecosystem for creators, platforms, web users and crawls will be better when crawling is more transparent and monitoring, and Cloudflare’s efforts is a step in the right direction for everyone,” Steve Hoffman, CEO of Reddit said in a statement.

When asked about the Cloudflare advertisement, Openai said that its Chatgpt model aims to help connect its users to the content on the web, similar to search engines, and that it merged the search for its chat functions. The company also said it used a separate model on what Cloudflare suggested to allow publishers to refer to how artificial intelligence crawls, known as Robots.txt. Openai said that the Robots.txt model is already works and unnecessary Cloudflare changes.

Tighten training data from the war

Artificial intelligence models require a tons of data for training. This is how they can provide detailed answers to questions and do a decent work (if it is incomplete) in providing a wide range of information. These models feed on incredible quantities of information and lead to contacts between words and concepts based on what you see in these training data.

The problem is how developers got this data. There is now Dozens of lawsuits Between content and artificial intelligence firms. Two major judgments witnessed only last week.

in One caseA federal judge spent Anthrobra followed the law when he used books protected by copyright to train the Claude-to the concept called fair use. At the same time, the judge said that the company’s creation of a permanent library of books was not, and he ordered a new trial on these piracy allegations.

In a separate case, a judge ruled In favor of the definition In a dispute between the company and a group of 13 authors. But Judge Vince Chapria said that the ruling in this case does not mean that future issues against the definition or other artificial intelligence companies will go in the same way. Basically, “these prosecutors presented the wrong arguments and failed to develop a record to support the right record.”

The idea of ​​charging reptiles to visit a site is not completely new. Other companies, such as TollbitProviding services that allow web owners to impose artificial intelligence companies on crawling. Allen, President of Control Ai, the privacy and media in Tollbit, said the environment surrounding this technology is still developing. “We believe it is very early for the content market to be formed, and we have just started experimenting here,” CNET told CNET. “We are excited to see many different models flourish.”

Imad Khan from CNET contributed to this report.