Theory Tuesday- Statistics’ Place in Big Data

Interesting, but long, talk about statistics place in the Big Data world:

I’d suggest watching from about 10 minutes in to about 40 minutes.

“Statistics”, “data mining”, and “bioinformatics” are all on the decline according to Google Trends, while “Big Data” is booming. Many big data people don’t see the need for statisticians because of their seemingly antiquated/belligerent/unhelpful opinions on model validity, result confidence, and experiment design. However, people who ignore statistics are condemned to re-create statistics.

In my experience, the people who don’t see value in statistics are action-oriented and typically mathematically-ignorant. These people want to do something, and they are not especially interested in how accurate their actions are. More responsible big data teams will be built with people with three skill sets: programming, math/statistics, and domain knowledge.

Leave a Reply

Your email address will not be published. Required fields are marked *