My Reddit AMA on the Law of Web Scraping

Part of why I find the topic so fascinating is that I have complicated feelings about scraping from a policy standpoint.

On the one hand, I believe property owners should generally have the ability to decide how their property is used. Scraping can involve unwanted and unconsented uses of a property owner’s property, in ways that can be pernicious.

On the other hand, I believe in access to information. (My mom was a librarian). To the extent that scraping enables better insights, better knowledge, and stronger markets, it’s a social benefit.

Furthermore, the Internet succeeded in part because of its seamless, low-friction, low-transactions-cost navigation. To keep that architecture, browsing the Internet should be a permission-less experience. Anti-scraping doctrines threaten to convert web browsing into a permissioned experience, where there are checkpoints at every site’s perimeter and permission is needed for access. Those barriers threaten to undermine the Internet’s navigability and utility, in ways that will disadvantage all of us.

[Note: Trespass to chattels doctrines are just one way that services can impose permissioned access into web navigation. Copyright law is another. Yet another is the online age authentication mandates sweeping the globe. For more on my opposition to age authentication mandates, see https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5208739 An obvious point: if age authentication checkpoints become ubiquitous in the Internet, that will have major impacts on scraping because most content will live behind age authentication walls].

I have never fully written up my complicated thoughts about these topics, but this piece will give you a sense of my thinking. https://blog.ericgoldman.org/archives/2013/04/rethinking_onli.htm

Here’s how I frame some policy considerations of scraping for students:

On the plus side: Scraping can enable services that create significant social benefits, such as search engines, price comparison bots, and AI modelbuilding. Also, anti-scraping efforts are frequently anti-competitive, i.e., an intent to manufacture or maintain monopolies. In particular, anti-scraping tactics can represent an attempt to obtain IP-like rights, even if the service doesn’t qualify for IP protection. You can see some of these dynamics in the Register.com v. Verio case, where Register.com was using contracts and trespass to chattels to protect non-copyrightable material (WHOIS registrant contact info) to preserve its first-mover-advantage control over that contact information.

On the minus side, scraping can look like a DOS attack (intentionally or not). It can facilitate massive copying, including possibly copyright infringements. And scraping can be a massive invasion of consumer privacy.

So, from a policy standpoint, is scraping a good or bad thing? ¿Por qué no los dos?: Or, to put it into legalese, “it depends.”