Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square60fedilinkarrow-up11arrow-down10
arrow-up11arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agomessage-square60fedilink
minus-squareCynicus Rex@lemmy.mlOPlinkfedilinkarrow-up0·3 months agoLies, as in that it’s not really “blocking” but a mere unenforceable request? If you mean’t something else could you please point it out?
minus-squareDa Bald Eagul@feddit.nllinkfedilinkarrow-up0·3 months agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up0·3 months agoIs it a lie or a simplification for beginners?
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up0·3 months agoLie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up0·3 months agoIt’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squareReversalHatchery@beehaw.orglinkfedilinkEnglisharrow-up0·edit-23 months agoIs it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up0·3 months agoMaybe? But it’s not like that’s the only alternative thing to say, lol
minus-squaremox@lemmy.sdf.orglinkfedilinkarrow-up0·3 months agoAssuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.
Lies, as in that it’s not really “blocking” but a mere unenforceable request? If you mean’t something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.
Is it a lie or a simplification for beginners?
Lie. Or at best, dangerously wrong. Like saying “Crosswalks make cars incapable of harming pedestrians who stay within them.”
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Assuring someone that they have control of something and the safety that comes with it, when in fact they do not, is well outside the realm of a simplification. It’s just plain false. It can even be dangerous.