You’d probably feel differently if it were your service. Should you be able to control who scrapes your sites or should that be all or nothing?
For the record, I fucking hate what the internet is becoming. I naively believed that even if shit got cordoned off into the walled gardens that are mobile phone apps, the web would remain as open as it was. This is a terrible sign of things to come.
No, I wouldn’t feel differently. In fact letting search engines scrape and point to your content is what leads people to your site. It’s free advertising. If you’re going to let one search engine in, you should let them all in. If you want to be public, be public. Otherwise put up a login firewall and go private.
It’s not just search engines. Lots of people on Mastodon were using robots.txt to block ChatGPT (and any other LLM company they knew of) from scraping their sites/blogs.
I disagree, to a point. I want to be able to control my services to the greatest extent possible, including picking who scrapes me.
On the other hand, orgs as large as Google doing this poses a real threat to how the internet works right now which I hate.
You’d probably feel differently if it were your service. Should you be able to control who scrapes your sites or should that be all or nothing?
For the record, I fucking hate what the internet is becoming. I naively believed that even if shit got cordoned off into the walled gardens that are mobile phone apps, the web would remain as open as it was. This is a terrible sign of things to come.
No, I wouldn’t feel differently. In fact letting search engines scrape and point to your content is what leads people to your site. It’s free advertising. If you’re going to let one search engine in, you should let them all in. If you want to be public, be public. Otherwise put up a login firewall and go private.
It’s not just search engines. Lots of people on Mastodon were using robots.txt to block ChatGPT (and any other LLM company they knew of) from scraping their sites/blogs.
I disagree, to a point. I want to be able to control my services to the greatest extent possible, including picking who scrapes me.
On the other hand, orgs as large as Google doing this poses a real threat to how the internet works right now which I hate.