LinkedIn: It’s illegal to scrape our website without permission
I&#x27;m following this story with a lot of interest. I&#x27;ve done (and still do!) a lot of data crawling&#x2F;scraping. In the past I&#x27;ve worked on so-called &quot;alternative data&quot; collection and analysis for financial forecasting.<p>Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of satellite imagery for oil tankers or manufacturing centers, but it can also come from the very creative use of scraped and aggregated data. It&#x27;s typically very difficult to identify, collect and analyze on a technical level (as &#x27;chollida1 has lamented in the past: normalization, labeling&#x2F;bucketing and analysis of disparate data across different formats, sources and processing timeframes is a pernicious problem at this scale). From a compliance standpoint there are also generally strict requirements governing legality of use.<p>Depending on the specific data, you might be capable of predicting earnings or broader market movements with a &lt;5% margin of error each quarter for years at a time (I&#x27;ve personally seen and worked on projects with &lt;1%, but that&#x27;s the exception, not the norm). That tactic is usually found at discretionary funds; at quantitative funds the uses are much more abstract and cross-pollinated so as not to target single-equities, but rather holistic trends. Regardless, every fund is using data in some way these days; it&#x27;s just a matter of how sophisticated, creative abstract they get in their analysis of it.<p>hiQ Labs doesn&#x27;t collect data for this specific purpose, but it is absolutely related. In the past I have stayed away from crawling LinkedIn and Yelp precisely because they are very litigious (regardless of the eventual outcome and legality). Now that there&#x27;s another relatively high profile case out in the open like this, I&#x27;m interested in seeing how it proceeds and what the ramifications will be for companies that collect data across a wide range of uses. As Grimmelman mentioned in the article, this can impact a lot of types of businesses, not just those in the same space as hiQ. Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data.