So it appears it’s possible to predict movement in the Dow Jones Industrial Average.  Serious technical stuff about this is here:

http://www.nature.com/srep/2013/130425/srep01684/full/srep01684.html

Less scholarly stuff, pretty much all over the place.

As you know, here at Big Data and the Law we are not in the simple reposting business.  We’re here to ask questions, offer opinions and occasionally be alarmist.  This is post is a little alarmist and a lot opinion.

First the good news.  The market predicting was accomplished using publicly available data from Google Trends.

That’s it.  No more good news.

Now the bad news.  Certainly there are other data that can be used in making market forecasts.  No doubt the Google Trends data used in this study can be replaced with something else – or used in conjunction with other stuff.  There is no reason to assume that such other data will be publicly available.  Google has data that is not published in Google Trends, and Google can stop publishing Google Trends at any time.  Data holders now have the power to choose winners and losers merely by determining who they give access to their data.  Consider the Twitter Fire Hose – PeopleBrowsr dispute.

Further, in addition to data, market predicting requires computational resources and skilled data analysis professionals.  The necessary resources and skilled people are unlikely to be accessible to the small investor.  It is conceivable that less wealthy individuals could come together, pool their resources and get close to what the super-rich can accomplish – perhaps through some subscription model.  I doubt it though.  Money always has the advantage.  The rich do in fact get richer.  Look it up.

If you’ve read the recent posts here about social media and the publication of non-public market information (which you should – because they’re really great), then you know how I feel about keeping open access to information that might affect your investment decisions.  Addressing that problem is orders of magnitude easier than addressing this prediction thing.

This is just one of many areas in which we will need to reevaluate how Big Data relates to the fundamental presumptions we have about law and society.  Collecting and using data (if the data are publicly available) does not, of itself, conflict with the notion of fairness our markets are supposed to be built.  However, the scale of Big Data creates new issues that can’t be ignored.  We have to find ways to preserve the benefits of technology without technology undermining the rules of law and society that we have agreed to live by.

I don’t have much faith in the ability of government (neither regulators nor legislators) to keep up with this, or in some cases to understand it.  (Watch C-Span sometime.  That should frighten you.)  I’m a little more optimistic about the judiciary though.  Judges have shown some very good technology related insight, and they have greater freedom of action than do regulators and legislators.

Over time we’ll talk about what we might do about this sort of thing.  In the meantime, I think we need to work together to counter the media’s apparent lack of interest in asking questions.  Please comment on articles you see that miss the point.