"Black swan" is a catch-all phrase for "outliers" or wildly unexpected events and processes: something such as the 2008 banking crisis, 9/11, the rise of Google et al.
There is an inherent danger that Big Data Analytics with algorithms galore, machine learning and predictive forecasting may fall foul of Black Swans. It may rely on the "narrative fallacy", the way past information is used to analyse the causes of events when so much history is actually "silent". It is the silence - the gap - the missing energy in the historical system, which produces the black swan.
The rise of the "citizen data scientist" could, I suggest, be the antidote to such dangers. Could Operational and line of business workers (with their deep understanding of cause and effect on the front-line) be such practitioners? -
- Nurses and doctors on hospital wards
- Field-service engineers attending offshore turbine farms
- Quantity Surveyors on construction sites
- Loss-adjusters at accident sites
If BI and Analytics moves beyond beyond just presentation and visualisation to being intuitive self-service BI it can support a broad range of "citizen data scientists".
- Information Consumers
- Information Creators
Consumers need standard KPI reporting but with the ability to personalise and re-prioritise to reflect daily requirements
Creators need to test theories and ideas, quey data and author new dashboards and reports- and share them of course
Analysts want to visually spot outliers and see if they are indeed random and unrepresentative or threatening black swans.
That way you can apply an optimal mix of citizen and full-blown data scientists to gain real competitive advantage,
The shortage of data scientists and the resulting high salaries they command is giving rise to new self-service tools, automating all stages of data science so business analysts, marketing managers, IT staff and others could perform advanced analytics as part of their jobs. By 2017, Gartner says, the number of these citizen data scientists in small and large organizations will grow five times faster than the number of highly skilled data scientists. Forrester agrees that the “huge demand” for data scientists will not be met in the short term, “even as more degree programs launch globally.” And the demand for advanced data analysis will only increase in the coming years with the rise of the Internet of Things.