Product Analytics: are you doing it wrong?

Shipra Mahindra

Is your Product Team making these common mistakes when it comes to tracking and analysing data? Here’s how to identify and fix them.

Every Product Manager knows the importance of tracking interactions with your product. Keeping an eye on data is essential so you can improve customer experience and measure how you’re tracking towards your KPIs. However, it’s not always easy to understand which metrics are the right ones to track.

Product expert Corinna Stukan recently spoke at a Brainmates Product Meetup to share tips on product analytics.

Corinna shared Product leader Marty Cagan’s expectations that, “Any capable Product Leader is expected to be comfortable with data, and to understand how to leverage analytics to learn and improve quickly.”

Despite this statement, a recent study from Mixpanel showed that less than 40% of product teams can validate their most important metrics with data.

The study also found that there isn’t a very high level of confidence among Product Managers when it comes to understanding conversion rates and knowing where users drop off. And around half of the people who responded said they weren’t confident in their ability to measure features after launch.

“It’s pretty clear that we don’t have enough resources and enough training about this topic in the product community,” says Corinna, who is Vice President of Product at digital consultancy business Roam.

To rectify this, Corinna shared the most common analytics pitfalls she has experienced, as well as her tips for knowing which metrics to keep track of and why.

For better analytics, move beyond vanity metrics

A huge pitfall of analytics is that we often lie to ourselves and look at basic ‘vanity’ statistics like the number of sign-ups or free trial registrations.

Getting new members is great, but it’s only a very small part of your product’s picture. The same can be said about figures like time spent on page (unless you’re Netflix or a publisher), number of downloads or number of page views.

While these numbers on their own might look great, it is important to look deeper and analyse figures which have a deeper context and in turn drive behavioural change and product updates. “We’re always talking about being data-driven, but what most of us are actually doing is being data-informed,” says Corinna.

“What we really need to ask ourselves beyond these ‘vanity metrics’ is; do our users actually activate and get value from the product? Do they engage with it and do they keep coming back?”

The numbers you look at should also reflect your goals for the product itself and not, for example, results like Facebook ad click-throughs, which the marketing team is more directly responsible for.

Don’t over-complicate things or track too much

 Corinna referenced a past product she worked on, where the tracking list consisted of over 300 separate data events.

“We create all these complex event names which only a few people understand. As a result, it’s hard to share and discuss the figures.”

Instead of having data overload and an endless list of figures that are hard to understand, what’s important is to ‘democratise’ by creating events and reports everyone can understand, then sharing them frequently and openly across the team.

Another problematic behaviour is ‘over tracking’ and keeping an eye on every single statistic. “You end up with an endless list of events which doesn’t get looked at regularly,” Corinna explains. “When a funding round or presentation opportunity comes up, you scramble for useful insights—and usually only look for the positive ones,” she says. “It’s not helpful for learning about product usage behaviour.”

An additional risk of over tracking is blowing your budget. “A lot of product analytics tools or data aggregators charge based on monthly volume of events. And while I always advocate for a good setup of tools, you might be racking up some costs that you don’t have to.”

Corinna says it’s fine to start with as few as 20 metrics on an MVP. These numbers should tell you the most important things about how users interact and engage with your product.

Identify ‘actionable metrics’

Corinna recommends creating a metrics checklist and always double-checking if the figures can really be used to drive actionable behaviour change.

“Your figures need to be comparable,” she explains. “If in doubt, use a ratio or a percentage of something instead of a total number so it can be compared.”

Consider that when you look at the success of a company, you take profit into account, not just total revenue. When reviewing a product’s performance, “You look at how much it sells for, how much it costs to make the product and how you calculate the profit margin.”

“Profit margin is actually a great example of a ratio that can be looked at over time. And it gives really good insights about whether you’re doing well or not,” Corinna says.

Instead of looking at total signups over a month to month period, an actionable metric is how many of those signups converted to paying customers or continued to engage with the product.

To create actionable metrics, you may also wish to:

  • Define what counts as an ‘active user’, instead of just tracking app launches, sessions or sign-ins
  • Look at retention and visitor frequency
  • Tie your metrics into your product’s goals or its ‘Product North Star’ so they can demonstrate how you are really tracking against goals and KPIs.

We’ve long been advocating for the close collaboration between product, design and tech in product teams but data still seems to live in its own silo.

Corinna’s final reminder on analytics is for Product Managers to take an active interest and do more than touch base with the Insights team once a month.

As Corinna explains, “Some of the best PMs I have spoken to didn’t have an analyst team so they just rolled up their sleeves and set things up themselves. Otherwise they found a way to include analysts on their teams.”

The same way you want your designers and developers to be part of your long term team so they can build their knowledge of the product over time, it makes sense for data scientists and analysts to have domain knowledge and expertise.

Include these people in your production meetings,” says Corinna. “Make them part of your sessions as a constant member of your product team. Share what’s on the roadmap and what kind of problems you’re trying to solve.”

Getting data analysts and insights teams more involved with your product will allow them to get context on the feedback loop and provide more useful insights.

And it’s all about sharing. As a Product Manager, you should be ensuring data events are easy to understand and that your dashboard is accessible and visible to everyone in your team. People should be able to discuss metrics as a way of confidently determining how your product is truly performing.

“Even if an ugly truth comes out because you have only been reporting acquisition numbers,” Corinna shares, “You have your product team there to solve problems and there is no point hiding them away.”

With the right approach to analytics, Product Managers can stop lying to themselves and access helpful data that can deliver better outcomes for all stakeholders.

What’s next?

Shipra Mahindra

Shipra Mahindra | Author

Shipra leads our Product Talks Auckland Meetup. She is passionate about using product management to make people's lives better. She brings together both her engineering & business expertise to ship continuous value to users and make an impact. She completed her MBA at UNSW after a previous life as a Software Developer and is currently enjoying solving problems at Xero.

Product Management Training