I completely disagree. The road to hell is paved with good intentions. What he proposes is just another flavor of technofascism, control disguised as ethics. These so-called humane scraping barriers will end up blocking humans, not machines. We have seen this before with CAPTCHAs, reCAPTCHAs, and all those “human verification” gimmicks, always bypassed by bots and always annoying to real people. Personally, I have nothing against crawlers and bots; they should do their shady jobs. Trying to wall yourself off is just meh. And those fancy licenses or “ethical use” terms won’t change anything either. The web is not the United States, and nobody really cares about someone’s imaginary social contracts. Maybe it’s time to accept a simple fact: once something goes on the web, it becomes public territory, and no one can still pretend to control the flow of information.
Fortunately this quixotic proposal is just some guy’s blog post.
I am seeing from these comments that my proposed solution was pretty naïve. I intended this blog post to be a sort of thought experiment to challenge assumptions made about the web pre-AI rather than my thought up technical solution to be the main focus.
I might go back and re-write some of this post to gear the focus more towards my main points of the social contract between bot and human shifting (especially with copyleft/ShareAlike), the web becoming less “open”, how this is not a new idea since the DMCA already considers standards for automated access, etc.
The ‘web scraping consent model’ as you so bootlickingly put it is simple: if you put something on the internet, anyone can and will look at it. If you don’t agree to that, then we’re better off without you.
It’s probably a little bit like when people used to copy tapes of their favourite artists in the 80s. And then that was just for private use and home projects.
But this is next level.
