I chucked in being one of the senior partners in my medium-sized practice 19 years ago and retired. One of the comments that was strongly behind that decision was made by one of my erstwhile partners during a partners’ meeting. It was (expletives deleted), ‘I don’t want to hear any more of your statistics, I know what is going on the practice’.
Now, for years, I thought the partner who ‘just knows what is going on’ – no matter what hard evidence there may be to the contrary – was the most dangerous person in the room. To be fair, they probably still are. They are like a pilot who won’t look at their instruments.
However, latterly, I am seeing another sort of person who I think is nearly as dangerous – the data wonk. Data wonks are the sorts of people who think the data tell you everything. And they are very dangerous, too.
There are several reason for this, but (sorry, this will get a bit philosophical) the main ones are:
Data are collected with a predetermined view of the important/permissible outcomes. People design data capture to collect the data they think are important. That involves a value judgment. I’ll give you an example from the world of forensic science. The tests the police choose to run are probably those which will give them evidence on which they can obtain a conviction. The way science is meant to work is to run tests which maximise the probability that the hypothesis (accused is guilty) will be refuted (i.e that the accused is not guilty). If that seems a bit arcane (the argument is developed further here http://www.criminallawandjustice.co.uk/features/Case-Defence-%E2%80%93-Bias-Forensic-Decision-Making-Process), then consider this…your webmasters/agency will be very happy if people spend longer on your website. They believe this measures ‘engagement’. It’s a 'good thing'. They may be right. On the other hand, if you have a confusing website that is hard to navigate (as many designed to be mobile friendly are), or lacks important information (like how to get in touch with the accounts section to pay your bill), they’ll also spend longer on your website, and that is a bad thing.
2. Data are collected over time. The choice of the time period can be critical. The large majority of data collection regimes are short-term in outlook. People want to tell you month by month, quarter by quarter how things are going. They design measures to yield results which change over the selected period.
However, important changes often happen on fairly long term timescales and the danger of an over-concentration on rapidly moving data is that business decisions are taken which are suboptimal in the long term because they make seem to sense short term.
Which is better, to improve a process 40% in 1 year then 1 % a year for 20 years or 5% a year for 20 years (compounded)?
3. Data measurements are always estimates. There are always roundings. When something which is 7.6% is presented as 8% it looks better than something which is 7.5% (7%). This is only important when the numbers are big, but…
4. The relative importance of data. One of the problems with data is that modern information systems allow huge amounts of data to be collected. This is where the data wonk becomes most dangerous because there is a tendency to regard data as valuable per se and the true data wonk will spend hours peering at data, much of which is of little significance….and that is the key issue. It is very easy to be seduced by the availability of data to lose one’s sight of the bigger picture. If you collect enough data, some random variance may well pop out that looks significant, which might well be the beginning of a wild goose chase – at best.
5. Covariance vs Causality. If item A causes both items B and C to change as it changes, there will be a correlation between the movements in the values of B and C…and it is VERY VERY easy to infer that they are causally related, which they aren't (this is how the tabloids make such silly claims about medical research). Confusing covariance with causality is normal, but it is wrong.
(If this is interesting, Ben Goldacre's 'Bad Science' will be a read you'll enjoy).
What to do? Well, the one thing I would do is to ask about any data point, what decisions it really helps you to drive and what the time-boundedness of those decisions will be. Then work out which data are predictive.
I am a numbers guy (MSc., chartered accountant), but I know numbers don't tell all the story.
By applying this sort of thinking in our business, we’ve been able to identify the core data that run our business and we no longer waste time looking at the rest. For example, we haven’t looked at Google analytics for years now.
The outcome of this is that we now spend a lot more time doing the 'up front; thinking and modelling BEFORE we decide what to do, so that the decision are made with high confidence levels. That's why, for example we don't spend anythign at all on SEO. We did the maths.
If all else fails, ask yourself ‘Does this make sense?’ Try to work out the real-world implications of the claims for the data. For things to measure anything real, there has to be a reason why the out-turn is as it is. Does the explanation offered make sense? Is it compelling? Are there other possible explanations? Have they been tested?
To reduce your firm's marketing costs, save staff time and grow your business faster, get in touch. 01392 423607 or see our websites at www.words4business.com , www.legalrss.uk (for law firms needing content) or www.myinfonet.com (for anyone needing a very fast, very simple, multi-modal communications/content management solution) and www.crosselerator.com or visit the legalRSS or MyInfoNet channels on YouTube.