Data-driven decision-making has become an accepted part of modern business practice. Organisations collect and analyse data to guide decisions. We form our hypotheses and perform experiments to validate them. We are all encouraged to “measure what matters”.
But if we are expected to accept that relying on data leads to better decisions, shouldn’t we expect to see the data that proves this is true? Intuitively it might seem obvious that objective data-driven decisions will lead to better outcomes, but it is our intuition that we are being asked not to trust.
Humans are fallible. We make decisions based on our judgements and biases. So, it stands to reason that using data to eliminate subjectivity from decision-making might be a sensible strategy. Even if we accept – as I will argue – that complete objectivity is impossible, maybe it is still a worthwhile goal? Unfortunately, through our efforts to achieve the impossible, not only do we ignore the importance of subjectivity in how we make decisions, but we also obscure the very biases that we should be examining more closely.
Data collection, analysis and interpretation is always the product of human perspectives. Before any data has even been collected, decisions have already been made on where to allocate resources and for what purpose. These decisions shape the strength of an organisation’s data capabilities and where it can focus its attention and have already begun to influence which decisions data will be used to support.
The people who collect data and perform analysis will have their own goals and be subject to influences within their organisation. Before a metric is presented to a decision-maker, many subjective decisions have already been made.
To quote an expert:
“People can come up with statistics to prove anything, … forty percent of all people know that.” – Homer Simpson
Ultimately the power to make decisions is also the power to decide how decisions are made. Regardless of how objective any analysis might be, if the data conflicts with the decision-maker’s perspective, they have the power to decide whether the evidence is sufficient for them to change their perspective.
Subjectivity cannot be removed from making decisions as a fallible, biased human always decides.
Is it possible to sidestep our human fallibility if we improve our analysis and interpretation of data?
You may be familiar with the term ‘statistical significance’ and its importance as a benchmark in scientific research, but even among experts it can be poorly understood, and its meaning is often misrepresented. Even so it is a standard rarely – if ever – applied to business decision-making.
We ignore the more rigorous requirements of research methodologies while still claiming to be performing “experiments”. If you introduce a feature and your performance metric increases, you might decide the value of the feature has been proven, but are the results significant?
When professional success is based upon getting certain results, it is far too easy to make decisions that lead to the desire outcome. In scientific research the failure to be able to reproduce the results of a significant proportion of published studies is referred to as the replication crisis. Even when objectivity is considered non-negotiable, and analysis is completed and supervised by experienced researchers, it is clear that results are still influenced by human biases.
Even if we had access to unbiased data and could perform objective analysis, many of the problems product leaders face are analytically unsolvable.
Let’s look at prioritisation as an example of a common task that might appear to be an opportunity for an analytical solution.
When prioritising product investment, we should consider:
With all these factors to consider, there is no one ‘correct’ way to calculate how each item should be prioritised. In mathematics, this is referred to as a combinatorial explosion – there are just too many variables!
Some problems simply cannot be solved algorithmically by applying steps in a pre-determined order – but the good news is that human beings are experts at solving problems anyway.
Consider how you would write software to determine the number of deer that will be at a selected waterhole tomorrow morning. You would need to factor in variables such as:
Perhaps some data can be excluded, such as the current position of Venus in the night sky – but we can’t know it can be excluded until we perform the calculation. Water pH, pollen count, days since the last earthquake? To calculate a solution, you must include it all.
There is no way your algorithm could ever reach a result by tomorrow morning, but a local hunter can probably give you an accurate estimate. This is not a result of the superior computing power of the human brain, but of a heuristic approach to problem solving.
We don’t know that the position of Venus can be excluded, but we do so without even considering it. Same with Mars.
We do the same thing when we apply our product intuition to prioritising investment.It is our ability to realise what is relevant that makes us generally intelligent, but it does put the final nail in the coffin of the idea of objective, data-driven decision-making.
The subjectivity and biases we apply in deciding what data we collect and incorporate in our analysis are not a fault in the process we should aim to eliminate – they are vital to our decision-making ability.
There is no doubt that subjective human decision-making can be deeply flawed. We make judgements based on limited personal experience, constrained by social norms, and influenced by base instinct.
There is also no doubt that by measuring and analysing the world we can improve our collective ability to make better decisions. However, by asserting that we can make our decisions objective using data, we obscure and ignore the subjectivity that remains.
We should always be conscious of the following:
Data is a powerful tool that we should use to improve our ability to make better decisions, less subject to bias and influence – but pretending to be objective only enables us to continue making subjective decisions behind a smokescreen of impartiality. I recall once spending several hours re-scoring initiatives so that an “objective” scoring system would prioritise them according to what a business leader wanted.
As product leaders, we must see both the truth of the world as it is and the possibilities of what it could be. We should embrace and encourage the use of data to help us to create value for our customers and our businesses, but we should also accept the daunting responsibility that we are the deciding factor – not the data we’re presented with.
Every day we make decisions based on assumptions without knowing if we’re right. But our power comes from our capacity to measure results and to update our assumptions, and then to try again.
You need a compelling product vision which describes the difference your products will make, and an actionable product strategy which maps the path forwards. These should never remain static. If analysis shows you that your underlying assumptions may be wrong, then this can’t be ignored, but when faced with uncertainty that no amount of analysis can eliminate, it is your vision and strategy that will allow your organisation to move forward with confidence.
If your product vision and strategy is not helping your teams escape the paralysis of analysis, then Brainmates can help. We partner with businesses to help them articulate their vision, define their strategy, and implement effective product metrics.
Organisations that invest in becoming product-led improve their value and performance – and we have the data to back that up. Reach out to us for an initial consultation.