Product Management in the Age of Generative AI

Product Management in the Age of Generative AI

By MARTIN NORTH

It might be a cliché, but the one constant in product management is that nothing is ever constant.

But the changes that are now visible on the horizon, driven by the sudden emergence of functional AI tools, are difficult to comprehend even for product leaders who have become accustomed to rapid technological change.

Generative AI platforms like ChatGPT, Bard, GitHub Copilot and Midjourney are the forerunners in a wave of platforms about to revolutionise how all digital assets are created and managed. While we are still only beginning to explore how AI can be utilised, as AI systems become increasingly capable of generating websites, software, video, music, and animation, many of the skills that have been the foundation of the technology industry for decades will become redundant.

This isn’t hyperbole; this is happening now. The industry may be unrecognisable a year from now.

Any professional whose career depends on applying acquired knowledge should consider how AI will impact their future, and any company whose business relies on utilising these capabilities should be doing the same.

If product managers are still required in the future, the teams within which they work and the roles they play will be vastly different. As product leaders, we must recognise the impact this revolution will have on product management and the industries in which we work.

AI-Driven Product Development

As AI tools for creating software become mainstream, the number of engineers required to create and maintain technology platforms will dramatically decrease.

Instead of written requirements being used to guide human engineers, they are being processed by AI and used to create working software.

Technical roles such as DevOps, quality assurance, security and delivery management will be able to be replaced by AI services operating with minimal human supervision. The ability for infrastructure to be managed as code will even allow AI to manage its own hardware requirements.

With large engineering teams no longer needed, Agile product development methodologies – such as Scrum and Kanban – will also become redundant. These frameworks are designed to improve the effectiveness of teams collaborating on software development.

Once development can be completed by AI tools, the problems they are designed to help solve will no longer exist.

The discovery and design phase of product development will consume most, if not all, of the time required to build a new product or feature. With fewer engineering constraints, the speed at which new products can be built will be limited only by the speed of their design.

Product roadmaps will become a relic of the past. When the desired attributes of a product have been decided, the development cycle will be nearly complete. The question of “when will it be ready to release?” will no longer be relevant. Once you know what ‘it’ is, most of the work will already be done.

Implications for SaaS Businesses and the Competitive Landscape

The growth of the SaaS segment has changed the way businesses purchase software. For enterprises, the decision to buy or build software has been trending towards buy, as it has become more cost-effective to purchase and integrate software services than to develop them in-house. Smaller businesses have gained access to tools they could never have developed themselves, now at a price they can afford.

However, SaaS businesses rely on the fact that software development is a specialised capability as the source of the value they create. The ease, speed, and affordability of software development brought about by AI will destroy the value proposition on which the SaaS industry is based.

Enterprises will be able to develop software in-house or purchase services, both for a negligible fraction of the current cost. Service aggregators such as Google and Microsoft will provide smaller businesses with complete suites of tools to run every aspect of their business.

As all software becomes essentially free, product differentiation will become a matter of choice.

If any feature a product team conceives of can be added to a platform immediately – and then just as quickly copied by competitors – the question is no longer whether something can be done, but whether it should be.

The only criteria to consider will be whether customers believe a feature adds or subtracts value.

The Future of Product Management

In the age of generative AI, product differentiation will be driven by strategic choices and a deep understanding of customer preferences.

Very small, highly-skilled teams will discuss ideas, select potential changes to test, generate updated software and monitor the results of experiments – with each iteration potentially lasting only hours.

Product squads may be replaced by small teams consisting of a product manager/marketer and one or more producers who use AI tools to create any required assets.

Some aspects of the role of the product manager may remain as they are today, as businesses will still need to understand their customers’ problems and anticipate how products can create value. Still, much of this work will be greatly streamlined through the use of AI to compile research, run experiments, and analyse data.

As development becomes faster and product differentiation based on features is no longer possible, the typical roles of product management and product marketing will likely merge, as positioning products within selected market segments becomes even more critical to product success.

As planning, resource management and roadmaps become obsolete, product managers will need to focus on curation – filtering the overwhelming number of possible functions and features available to create a unique and valuable offering in a carefully selected market.

It seems clear that the use of generative AI will bring about unprecedented changes in how digital products are developed and managed – but right now, this may be the only clear thing.

Now more than ever, product leaders need to be focused on the future – anticipating challenges and targeting opportunities – to create the successful products, and successful product-led businesses, that will inevitably emerge.

 

Brainmates has partnered with hundreds of organisations to solve their product challenges and help them adapt to new technologies.

Talk to Brainmates today.

Your Vision is Your Product

Your Vision is Your Product

By MARTIN NORTH

We spend our lives surrounded by products, and for those of us who have followed the path to managing and developing them, we use the label “product” to define what we do, and even who we are. However, there are many different ways to think about what “product” means, and whatever you think will impact how you approach product management.

Successfully collaborating within and across teams to improve our efficiency, efficacy, and effectiveness requires a shared understanding, but how can this exist if there is no agreement on what your product – or any product – actually is?

Brainmates’ Essentials of Product Management course defines a product as a combination of goods and/or services that satisfies a need for a specific market, is deliberately created for trade or commerce, and encompasses the entire customer experience.

It’s a rational definition that sets boundaries for our Product Management Framework, and it’s more than sufficient for most everyday practical questions related to managing product. But in a complex, messy and irrational world, a deeper understanding may require a broader perspective.

All products require at least two independent – but complementary – value propositions. Your customer must be willing to exchange something of value for your product, and you must be willing to sell it for what they are willing to exchange. Notice that this doesn’t require complete agreement. Your customer may have a very different view of the value your product creates.

Everyone involved or impacted by your product’s ecosystem will have a different perspective. The definition of product used above is still correct, but for even a single product, every aspect of the definition – “satisfies”, “need”, “market”, “trade”, ”customer”, or “experience” – will be understood differently depending on each participants perspective.

Your “product” means something different to every customer, investor, employee or competitor.
Where these viewpoints align – and where they don’t – can determine whether your product succeeds or fails.

When Brainmates partners with a new organisation, we like to begin by understanding their Product Vision. It’s an entry point to understanding how they view their products and the value they want to create. And whether your potential customers are aware of it or not, this is also their starting point.

If your Product Vision doesn’t tell a compelling story, this can undermine the effectiveness of every customer touchpoint. When a customer buys your product, they confirm that their vision aligns with yours – which first requires an implicit understanding of your vision.

Your product is a vision; a vision you create of the world that is slightly, or wildly, different than today, and that your customers can share. The more inspiring your vision is, the more likely you will be to create an ecosystem of customers, suppliers, employees and partners that are all aligned to create the future your vision describes.

Some obvious ways a lack of a compelling Product Vision can manifest include:

  • Focussing on product problems – where initiatives are designed to resolve problems with how your product will be used, without questioning why it will be used.
  • Focussing on business problems – when objectives refer to improving outcomes for the business, such as improving revenue, without connecting business objectives to creating customer value.
  • Focussing on competitors – where strategy addresses the features of competing products, trying to “catch-up” or “get ahead” without knowing the destination.
  • Focussing on platforms – when resources are invested in re-platforming, without questioning the logic of delivering low-value features more reliably.

Each of these patterns of failure (and many others) can result from a poor Product Vision, so every product leader should be confident that their Product Vision is strong enough to act as the guiding purpose for every product decision.

If your customers choose to align themselves with your vision, then it could also be argued that your vision is your product.
Of course, it is a combination of goods and services - but it is also an idea.

As a product leader, you tell the story of your product – the change it will offer your customers, how it will be created, and the type of organisation required to create it. These are all necessary ingredients for the narrative you are constantly reimagining and sharing. A con artist sells a vision without any underlying value. Authentic product-led organisations begin with a promise of change that inspires people to want to create that change. The more compelling your vision, the more successful you will be in attracting everything required to make it real.

If this doesn’t sound like your Product Vision, then the place to start is with a deep understanding of your customer and their problems, as well as your organisation and its capabilities. You must be able to imagine and communicate a better future for both that only your products can create.

 

Brainmates has partnered with hundreds of organisations to solve their product challenges, and this process often begins with facilitating the creation of a powerful Product Vision that can define the direction for a winning Product Strategy.

Talk to Brainmates today.

Unlock the power of Product-Led teams

Unlock The Power of Product-Led Teams

By MARTIN NORTH

With organisations debating and adopting competing strategies such as Sales-Led, Marketing-Led, or Customer-Led, learning about the benefits of being Product-Led might seem like an unnecessary distraction. How many different ways of being “led” do we need? Is this a way for product teams to argue that they should make all the decisions?

But Product-Led is different. Rather than choosing between opposing strategies, it seeks to find a balance. Rather than providing precedence to one function, it seeks to align all parts of the organisation. Rather than giving dominance to a single perspective, a commitment to being Product-Led represents a decision to balance the needs of customer and business, and in so doing, create value for both.

Extensive research clearly shows the costs of product failure and the benefits that flow to organisations that invest in becoming Product-Led. With 40% of new products failing, the cost of product failure is estimated at over $1 trillion per year. However, companies that become Product-Led have 2x enterprise value, a 1.8x growth rate and a 1.5x revenue multiple.

Product-Led seeks to minimise the risk of product failure and maximise the value that its products create. This does not mean all decisions are made within the product team, and this is not a grab for power by frustrated product managers. It means that all parts of the business transform how they think and perform to enable the business to put modern Product Management Practices at the core of how they drive value, not only with product teams but across all business functions. Product-Led organisations strike a balance between creating value for customers and value for the organisation.

The Three E’s

When considering organisational capabilities, one approach to categorising potential areas for improvements is using “the three E’s”:

  • Efficiency – doing more with less
  • Efficacy – achieving a planned outcome
  • Effectiveness – achieving the right outcome

When developing a product, you could aim to be efficient by utilising your resources wisely and creating output with minimum waste​. You could focus on being efficacious (a term used in medicine to describe a drug that does what it’s supposed to), making products that function as intended, free from defects​. Or your goal could be effectiveness – creating products that create value.

These are not exclusive choices, and it is essential that every area is addressed. We cannot succeed if we waste resources, if our products don’t work, or if they don’t create value. But there is a limit to how efficient or efficacious any organisation can be. Your waste can never be less than zero, nor can your defects or downtime.

There is no limit to how much value
you can create.

The power of being Product-Led is enabling teams that make products with minimum waste​, make products that work as expected, and make products that create value. If a typical development team costs over $2 million per year, then the cost of that team not being effective is all of that $2 million. On the other hand, every improvement we make to our effectiveness acts as a multiplier, converting investment in efficiency and efficacy into growth. The only pathway to sustained competitive advantage lies in maintaining efficiency and efficacy while striving to create greater value. The potential return on investment for being Product-Led is virtually unlimited.

Cycles Of Failure

Being Product-Led starts with the balance between customer and organisational value, but that is not where the balancing act ends​. In every area of product management, there are competing pressures that must be balanced​. If we fail to find that balance, we risk getting trapped in cycles of failure that destroy our effectiveness. However, traditional ways of solving problems force us out of balance.
 
A leader identifies a problem and declares “something must be done!”. The bigger the problem, the more extreme and less balanced the proposed solution. Unbalanced solutions lead to unintended consequences, as important areas are neglected. Problems begin to mount until a leader declares “something must be done!”. When looking for the root cause of any problem, the best place to look is usually the last solution. We need to break the cycle of failure​.
 
These are the six areas that Product-Led organisations need to focus on:
  • Strategy & Roadmap – developing a bold vison and strategy to deliver outcomes that are clearly linked to business strategy​.
  • Structure & People – creating the right organisational and product structure, and hiring and onboarding the right people with the required skills.
  • Practice & Process​ – implementing product management and capability frameworks across your team to achieve business outcomes.
  • Discover & Deliver​ – implementing continuous discovery and iterative delivery to solve valuable customer problems that also deliver business value.
  • Measure & Respond​ – measuring what matters and creating insights that drive decision-making in changing environments.
  • Align & Transform​ – aligning the entire organisation to balance customer needs with business benefits to achieve product growth.
While each of these areas is important, as your organisation grows and evolves, different areas will be more important for us to focus on at different times. We must balance our efforts across all areas, ensuring none are neglected. But we also need to find a balanced approach within each area. We have all observed cycles of failure that trap organisations​ and prevent us from finding balance. Here are some examples:

When We Fail to Balance Predictability and Adaptability

When developing Strategy & Roadmaps, we need to balance between predictability and adaptability​. If we overemphasise predictability, we force teams to adhere to rigid feature-based roadmaps and reward them for completing tasks instead of creating value.​ To correct this, we overemphasise adaptability. Teams set their own priority, but don’t align with each other or business strategy until someone says…​ “We need more predictability!”​.
 
To be effective, we need to build trust with customers and stakeholders over time, while also responding to changing circumstances. This is only possible if we successfully balance predictability and adaptability.

When We Fail to Balance Restructuring and Recruiting

When developing Structure & People, we need to balance between restructuring and recruiting​. When products are underperforming, we may restructure teams – creating new roles and areas of responsibility. However, if this is done without first understanding the capabilities required and gaps in existing capabilities, we create anxiety and frustration. Valuable team members forced to adjust to new responsibilities without support may decide to leave the organisation. ​

​To compensate for increasing attrition, we switch the focus to urgent recruiting. We recycle position descriptions to urgently find replacements, deliberately filling roles with people with the same capabilities as those they are replacing. Have you ever started a job and been introduced as “the new …” somebody? They are literally hoping you will be just like whoever you are replacing. But they were unhappy and left!​

After a brief period of enthusiasm, the existing issues reappear because they haven’t been addressed. With performance continuing to fail to meet expectations, inevitably the debate begins again about team structure.​

Restructuring teams and recruiting new team members both come at a cost. If we fail to balance the two, we reduce our effectiveness and prevent the expected benefits from being realised.

When We Fail to Balance Compliance and Flexibility

When developing Practices & Processes, we need to balance between compliance and flexibility​. Over-emphasis on compliance results in rigid processes that can’t account for the differences between products and teams. Teams waste time attempting to comply with prescribed ways of working that don’t add value. Resistance to processes eventually undermines compliance. Procedures are abandoned to allow teams the flexibility to deliver as they see fit, but this leads to inconsistent resultsand inconsistent communication with stakeholders who will ultimately insist on greater compliance.​
 
If we fail to balance compliance and flexibility, we reduce our effectiveness as opportunities to create value are lost while teams fight to comply with processes that are not fit for purpose, or lack of process leads to inconsistent or misaligned outcomes.

When We Fail to Balance Research and Testing

When focusing on Discovery & Delivery, we need to balance between research and testing​. If we spend too much time doing research, we risk never delivering a product to market. Frustration with the lack of delivery leads to a strategy of building and testing. Instead of validating ideas through research, every idea is considered worth building with the assumption that it will then be tested​. Poorly designed and unfinished products that are being “tested” lead to pushback from customer-facing teams who are in direct contact with customers, and the emphasis returns again to research.
 
Efficiency, efficacy and effectiveness can all be impacted if we fail to get this balance right, as we waste effort on developing features that don’t add value, release poor quality products to market, or fail to fully understand customer problems.

When We Fail to Balance Data and Intuition

When developing our capabilities to Measure & Respond, we need to balance data and intuition. If we over-emphasise the use of data, we can get stuck in analysis paralysis – unable to make decisions where there is any uncertainty – or perform analysis to support decisions that we know have already been made. In order to move faster, we start by-passing requirements and working on pet projects with executive sponsors that are “obvious quick wins”.​ When these initiatives fail to achieve the desired outcomes, we impose new criteria for using data to make decisions.​

Failure to find a balanced approach to analysis reduces efficiency and effectiveness as we waste resources attempting to justify low-risk decisions – or adding features that don’t add any value – which analysis could have helped us avoid.

When We Fail to Balance Pragmatism and Principles

When focusing on Aligning & Transforming, we need to balance between pragmatism and principles.​ If we are too pragmatic, we start to compromise in advance, entering a state of learned helplessness. We keep working the same way without any expectation of success. ​But this level of frustration can’t continue indefinitely, so eventually, we decide that something has to change and take a stand on Product-Led principles. We insist that there is a “right” way to do things and a “wrong” way, arguing from ideology without considering our organisation’s unique context.​ Rival camps form around alternative approaches, and delivery slows. In a bid to unblock decision-making, leadership forces through compromises. With our influence diminished, the status quo is re-established, and we go back to our old ways of working, still believing there is a better way.

If we fail to balance pragmatism and principles, not only do we fail to increase our effectiveness by being Product-Led, but by reducing our credibility and influence, we make it less likely that we can achieve these benefits in the future.

Breaking The Cycle To Become Product-Led

How many of these cycles of failure have you experienced? Maybe all of them? To become Product-Led, we need to break these cycles​. We need to stop reacting to issues with simplistic solutions that have already been tried and failed​. Instead, we must enable balanced approaches to solving problems that allow teams to become Product-Led​. Here are some suggestions for steps you can take in each area to escape the cycle:

Outcome-Based Planning

To balance predictability and adaptability, we need to implement outcome-based planning. Outcome-based planning provides predictability on what improvements the team is building towards while leaving the details of the solution to be developed iteratively through continuous discovery. We maximise effectiveness by ensuring customers trust us to deliver value consistently, not just occasionally, and allowing plans to adjust as teams perform discovery or when circumstances change.

Planning is a promise – “give us x resources and we will deliver y result”, but no one should expect a blank cheque ​in constrained economic conditions. If the sole purpose of the Product-Led team is creating customer and business value, that is what should be on the plan.

You must connect value to product
outcomes and communicate a plan
to achieve that outcome - this is your roadmap.

There are many good resources on outcome-based planning online, but if you understand the balance that this approach is trying to create, you are much more likely to be successful.

Capabilities Assessment

To balance restructuring and recruiting, we need to assess gaps in capabilities. By understanding the capabilities you have and the capabilities you need, you can have a transparent conversation about where to focus development and what skills you should be looking to recruit​. When we invest in restructuring teams or recruiting new team members, a better understanding of current and required capabilities will ensure a return on this investment in the form of increased effectiveness.

If you are using the same generic requirements to define multiple product roles, it may be that you haven’t considered what capabilities each role actually requires​. But no matter how clearly we define the requirements for a role – we fill that role with a person who will have their own unique strengths and weaknesses​. Developing capabilities should be a focus for every organisation, but recruiting will always be necessary to grow teams and acquire new capabilities.​

You must start with an environment of safety and trust – if you don’t have this, this should be your ONLY focus​. Within this environment, your current capabilities need to be assessed. An assessment tool such as the Association of Product Professionals Proficiency Framework can be used to identify strengths and areas for development​.
Once you understand the capabilities you have and the capabilities you need, you can have a transparent conversation about where to focus development and what skills you should be looking to recruit​.

A Shared Commitment to Adopting Practical Frameworks​

To balance compliance and flexibility, we need to gain shared commitment to adopting practical frameworks. By getting buy-in from all stakeholders on frameworks that target known issues and can feasibly be implemented, we improve outcomes without creating an unnecessary burden for our teams​. The right tools implemented in the right way boost effectiveness through consistent performance, properly aligned with strategy, while avoiding the frustration created by unnecessary bureaucracy.

If you’re not using any product framework, pick a problem area and review common approaches – you won’t have any trouble finding potential solutions​. If you are using frameworks – are they actually helping to create value? Do all stakeholders agree on the value the framework is adding?​

​Start with a single area of practice and recommend a fit-for-purpose framework and lean processes and get commitment from everyone involved​.

Don’t assume the tool is the issue. If you love how you work but are having trouble with scale – then a product management platform is a worthwhile investment. But if there is a reluctance to use a framework or it’s not giving stakeholders what they need – a new SaaS platform will not solve the problem.​

​Using any framework requires ongoing realignment and coaching – make sure everyone understands why a framework is being used and how to use it, but if something is not working, adapt the framework for your needs​. Avoid the latest fad or changing just for the sake of changing. This will only reinforce the cycle of failure we’re trying to break.​

Continuous Discovery

To balance research and testing, we need to implement continuous discovery processes that include both. By breaking the discovery into small steps and progressing from research to testing as we learn, we avoid assuming customers can always tell us what they want or wasting resources building features no one values​.

Effectiveness and efficiency are increased by delivering products that create value, without over-investment in either
research or development.

Research and testing are both valid ways to reduce uncertainty and risk. The goal is to learn just enough at each step to justify continuing.​ Research could be as simple as talking to a small number of customers to possibly invalidate an idea, but like everything, this has diminishing returns. There is only so much you can learn from asking questions.​

You learn more from testing real products with real customers – but this is more costly.​

If you have one big proposal with one big price tag, then you need to spend a lot to justify the decision. And once you have already invested, there is a risk you might keep chasing your sunk costs, trying to validate the idea despite what your research is showing you.​

Break the investment into small steps of increasing size, and then validate only enough before each step to justify the next investment, and progress from research activities to product testing so that you don’t wait too long to collect real observations.​

Data Analysis to Reduce Risk

Product outcomes cannot be determined in advance – only predicted – so some uncertainty is inevitable.​ Using data as a tool to reduce risk focuses our attention on where there are unacceptable levels of uncertainty that require further analysis, and where the level of risk is insufficient to justify analysis paralysis.​ This ensures data resources are used efficiently only where they are expected to add value, and improves effectiveness by reducing the risk of investing in development that cannot be justified.

You must know your organisation’s appetite for risk, because if you are proposing an idea that is outside of this appetite, you are very unlikely to be successful.​ ​
Because uncertainty cannot be eliminated, we need to be realistic about how much extra certainty is required for a decision to be approved, and what the cost of gathering that data will be. There is no point wasting time and effort performing analysis to support a decision if the result is still likely to be outside of your organisation’s acceptable risk.​​

We need to avoid thinking that decisions can only be made if supported by data analysis. There is nothing wrong with using our judgement. It is unavoidable. We use our judgement when deciding whether we agree with what our data analysis reveals. ​

Adapting Product-Led

To balance pragmatism and principles, we must adapt Product-Led principles to each unique context. By knowing when to be an evangelist and when to be a pragmatist, you will build your influence as a knowledgeable expert who also knows how to get things done.​​ The potential benefits of being Product-Led will be realised gradually with less risk or resistance or conflict.
 
Invest time to understand frameworks. They’re not all worthless, and there is no one single “best” approach.

You have to develop your own principles
in order to decide when to
compromise them.

Think in advance about why any principle may not apply to your context.​ ​

Always start with what you believe the “right” way is. Even if you later compromise, you should always begin with a clear explanation of your preferred approach.​ Then focus on feasibility and practicality – ultimately, the “right” way is the way that works. ​

​Don’t try to boil the ocean – you can’t change everything at once.​

If you strongly believe in a particular approach but anticipate resistance, use your relationships with stakeholders to surface potential objections and find potential allies as early as possible.​ If your viewpoint doesn’t prevail, commit 100% to the adopted approach. Undermining the outcome by not offering your full support will not increase your influence on the next decision.​

If your approach is adopted, and other stakeholders resist its implementation – this is also your problem. Again – the only “right” way is the way that works. If an approach fails due to lack of support – then it was the wrong approach.​

Self-Authorise and Start

In order to begin, we may need to authorise ourselves to step beyond the perceived limits of our formal authority. In modern organisational structures, those with the greatest potential to lead change are often not those in positions of authority. If this is you, then you need to give yourself permission to act. In the face of resistance, apathy or the lack of an immediate mandate, you need to self-authorise.​ This needs to come from a place of self-awareness – you must know your skills and limitations – but with this knowledge, anyone in any role can lead change. Leadership describes a way of acting, not your place in a hierarchy.

Being Product-Led enables teams to create customer and business value, but this requires balance in all areas.​ Traditional problem-solving forces teams into cycles of failure where balance is impossible, and how unlocking the power of Product Led teams means breaking the cycles and allowing teams to find balance.​

Take a look at your organisation and look at the areas where you just can’t seem to make any progress. Are you trapped in any of the cycles of failure in this presentation? Or is there another cycle you can see? ​Instead of repeating the same mistakes, think about how one of the balanced actions we’ve looked at today might help you take a first step to become Product-Led​.

Data Doesn’t Make Decisions – People Do

Data Doesn't Make Decisions - People Do

By MARTIN NORTH

Data-driven decision-making has become an accepted part of modern business practice. Organisations collect and analyse data to guide decisions. We form our hypotheses and perform experiments to validate them. We are all encouraged to “measure what matters”.

But if we are expected to accept that relying on data leads to better decisions, shouldn’t we expect to see the data that proves this is true? Intuitively it might seem obvious that objective data-driven decisions will lead to better outcomes, but it is our intuition that we are being asked not to trust.

Humans are fallible. We make decisions based on our judgements and biases. So, it stands to reason that using data to eliminate subjectivity from decision-making might be a sensible strategy. Even if we accept – as I will argue – that complete objectivity is impossible, maybe it is still a worthwhile goal? Unfortunately, through our efforts to achieve the impossible, not only do we ignore the importance of subjectivity in how we make decisions, but we also obscure the very biases that we should be examining more closely.

A Matter of Perspective

Data collection, analysis and interpretation is always the product of human perspectives. Before any data has even been collected, decisions have already been made on where to allocate resources and for what purpose. These decisions shape the strength of an organisation’s data capabilities and where it can focus its attention and have already begun to influence which decisions data will be used to support.

The people who collect data and perform analysis will have their own goals and be subject to influences within their organisation. Before a metric is presented to a decision-maker, many subjective decisions have already been made.

To quote an expert:
“People can come up with statistics to prove anything, … forty percent of all people know that.” – Homer Simpson

Ultimately the power to make decisions is also the power to decide how decisions are made. Regardless of how objective any analysis might be, if the data conflicts with the decision-maker’s perspective, they have the power to decide whether the evidence is sufficient for them to change their perspective.

Subjectivity cannot be removed from making decisions as a fallible, biased human always decides.

Can Subjectivity Be Avoided?

Is it possible to sidestep our human fallibility if we improve our analysis and interpretation of data?

You may be familiar with the term ‘statistical significance’ and its importance as a benchmark in scientific research, but even among experts it can be poorly understood, and its meaning is often misrepresented. Even so it is a standard rarely – if ever – applied to business decision-making.

We ignore the more rigorous requirements of research methodologies while still claiming to be performing “experiments”. If you introduce a feature and your performance metric increases, you might decide the value of the feature has been proven, but are the results significant?

If we ignore this question, it is far too easy to claim positive outcomes as wins,
while explaining away unexpected results as aberrations.

When professional success is based upon getting certain results, it is far too easy to make decisions that lead to the desire outcome. In scientific research the failure to be able to reproduce the results of a significant proportion of published studies is referred to as the replication crisis. Even when objectivity is considered non-negotiable, and analysis is completed and supervised by experienced researchers, it is clear that results are still influenced by human biases.

Some Problems Have Solutions, But Cannot Be Solved 

Even if we had access to unbiased data and could perform objective analysis, many of the problems product leaders face are analytically unsolvable.

Let’s look at prioritisation as an example of a common task that might appear to be an opportunity for an analytical solution.

When prioritising product investment, we should consider:

  • All the proposed initiatives
  • All the variables that might impact the cost of implementation
  • How they might be perceived by current or future customers
  • How competitors or regulators might react

With all these factors to consider, there is no one ‘correct’ way to calculate how each item should be prioritised. In mathematics, this is referred to as a combinatorial explosion – there are just too many variables!

Some problems simply cannot be solved algorithmically by applying steps in a pre-determined order – but the good news is that human beings are experts at solving problems anyway.

Consider how you would write software to determine the number of deer that will be at a selected waterhole tomorrow morning. You would need to factor in variables such as:

  • The local deer population
  • Deer group dynamics
  • Expected weather patterns
  • The behaviour of any predators

Perhaps some data can be excluded, such as the current position of Venus in the night sky – but we can’t know it can be excluded until we perform the calculation. Water pH, pollen count, days since the last earthquake? To calculate a solution, you must include it all.

There is no way your algorithm could ever reach a result by tomorrow morning, but a local hunter can probably give you an accurate estimate. This is not a result of the superior computing power of the human brain, but of a heuristic approach to problem solving.

We apply pre-determined rules to judge what is relevant and what is not to reach practical conclusions and avoid a combinatorial explosion.

We don’t know that the position of Venus can be excluded, but we do so without even considering it. Same with Mars.

We do the same thing when we apply our product intuition to prioritising investment.It is our ability to realise what is relevant that makes us generally intelligent, but it does put the final nail in the coffin of the idea of objective, data-driven decision-making.

Bias Is Necessary – The Risk Is Ignoring It

The subjectivity and biases we apply in deciding what data we collect and incorporate in our analysis are not a fault in the process we should aim to eliminate – they are vital to our decision-making ability.

There is no doubt that subjective human decision-making can be deeply flawed. We make judgements based on limited personal experience, constrained by social norms, and influenced by base instinct.

There is also no doubt that by measuring and analysing the world we can improve our collective ability to make better decisions. However, by asserting that we can make our decisions objective using data, we obscure and ignore the subjectivity that remains.

Before we measure what matters, we must decide what matters.

We should always be conscious of the following:

  • Who is making the decision?
  • Who benefits from their actions?
  • Whose advice are they listening to?
  • Have any parties that might be negatively impacted been considered?

Data is a powerful tool that we should use to improve our ability to make better decisions, less subject to bias and influence – but pretending to be objective only enables us to continue making subjective decisions behind a smokescreen of impartiality. I recall once spending several hours re-scoring initiatives so that an “objective” scoring system would prioritise them according to what a business leader wanted.

As product leaders, we must see both the truth of the world as it is and the possibilities of what it could be. We should embrace and encourage the use of data to help us to create value for our customers and our businesses, but we should also accept the daunting responsibility that we are the deciding factor – not the data we’re presented with.

Every day we make decisions based on assumptions without knowing if we’re right. But our power comes from our capacity to measure results and to update our assumptions, and then to try again.

You need a compelling product vision which describes the difference your products will make, and an actionable product strategy which maps the path forwards. These should never remain static. If analysis shows you that your underlying assumptions may be wrong, then this can’t be ignored, but when faced with uncertainty that no amount of analysis can eliminate, it is your vision and strategy that will allow your organisation to move forward with confidence.

 

 

If your product vision and strategy is not helping your teams escape the paralysis of analysis, then Brainmates can help. We partner with businesses to help them articulate their vision, define their strategy, and implement effective product metrics.

Organisations that invest in becoming product-led improve their value and performance – and we have the data to back that up. Reach out to us for an initial consultation. 

Get a Quote

We’ll get in touch with you to discuss how we can help your team.