TikTok parent company ByteDance is amassing huge volumes of web data way faster than the other major web crawlers
ByteDance may be planning to release its own LLM, and is aggressively using its web crawler, “Bytespider,” to scrape up data to train its models, Fortune reported.
Bytespider showed up on the scene in April, and since then, its rate of consumption puts web scrapers from OpenAI, Google, Meta, and Anthropic to shame.
Mashable Light Speed
Sam Crowther, CEO of Kasada, a company that specializes in bot management, told the outlet that Bytespider’s scraping rate is 25 times more than OpenAI’s GPTbot and 3,000 times the rate of ClaudeBot, which is Anthropic’s web crawler for its Claude LLM. Crowther also said that Kasada’s data has seen “huge spikes in scraping activity” from Bytespider in the last six weeks.
As Bytespider voraciously consumes the web, the U.S. government is trying to inhibit potential access of …