Technology Reddit threatens to block Google and Bing web crawlers

[Ink]

Content source

https://www.theverge.com/2023/10/20/23925504/reddit-deny-force-log-in-see-posts-ai-companies-deals

The Washington Post reported Friday that Reddit might cut off Google and force users to log in to Reddit itself to read anything, if it can’t reach deals with generative AI companies to pay for its data. Initially, Reddit seemed to deny the report. “Nothing is changing,” Reddit spokesperson Courtney Geesey-Dorr told The Verge, adding that the Post would soon be correcting its story.

But after the Post corrected that story, only one major detail had changed — the Post no longer suggests Reddit users would need to log in. The publication now writes that if Reddit can’t get AI to play ball, the company may block Google and Bing’s search crawlers, which means Reddit posts wouldn’t show up in search results.

 

[Seandc33]

First changing api keys so that third-party reddit apps would be forced to stop working, to now blocking search engines from indexing their search results.

Are they trying to run the site to the ground? Because they're sure doing a good job doing that.

 

[SumTingWong]

First changing api keys so that third-party reddit apps would be forced to stop working, to now blocking search engines from indexing their search results.

Are they trying to run the site to the ground? Because they're sure doing a good job doing that.

I think they are going to add and use Chinese search engines.

Isn't this one Chinese company acquired Reddit not too long ago?

 

[nicolaasjan]

The change is now official...
Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

Google is now the only search engine that can surface results from Reddit, making one of the web’s most valuable repositories of user generated content exclusive to the internet’s already dominant search engine.
If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn’t rely on Google’s indexing and search Reddit by using “site:reddit.com,” you will not see any results from the last week. DuckDuckGo is currently turning up seven links when searching Reddit, but provides no data on where the links go or why, instead only saying that “We would like to show you a description here but the site won't allow us.” Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward. Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google.

 

[Trident]

I am scared to use generative AI that’s been trained on Reddit. SCARED!

 

[oldschool]

Brave uses

Google fallback mixing​

For queries where Brave Search is not yet refined, your browser will anonymously check Google for the same query, mix the results for you and send the query data back to us so we can improve Brave Search for everyone.

Even though search.brave.com on other browsers like Edge doesn't have this setting, it still shows Reddit results (for all that critically valuable information @Trident ).

 

[cartaphilus]

I am scared to use generative AI that’s been trained on Reddit. SCARED!

Oh come on it's not that bad.

Just look at those wonderful news headlines written by code-davinci generative AI that was trained on reddit.

"Experts Warn that War in Ukraine Could Become Even More Boring."

“Budget of New Batman Movie Swells to $200M as Director Insists on Using Real Batman”

“Story of Woman Who Rescues Shelter Dog With Severely Matted Fur Will Inspire You to Open a New Tab and Visit Another Website”

“Phil Spector's Lawyer: ‘My Client Is A Psychopath Who Probably Killed Lana Clarkson’”

“Rural Town Up in Arms Over Depiction in Summer Blockbuster 'Cow#####ers'”

 

[lokamoka820]

I think Brave search still able to shows Reddit results.

 

[nicolaasjan]

I think Brave search still able to shows Reddit results.

Click on the slider bitton on the right and filter by "past week".
Search for e.g chrome reddit.
Result:

Then do the same with Google search.
More than 6 million results:

 

[lokamoka820]

Click on the slider bitton on the right and filter by "past week".
Search for e.g chrome reddit.
Result:

View attachment 284533

Then do the same with Google search.
More than 6 million results:

View attachment 284534

So what Brave shows was old results! Thanks for the clarification.