62: How to measure product manager performance

Early on in my blog, I wrote one of my most popular posts – 4 key ways to spot a successful product manager – about measuring the performance of product managers. The problem is that a lot – and I mean a huge amount – has changed in product management, and my own approach, since I wrote it.

I found myself describing to Martin Eriksson at his recent book launch some work I did at the UK’s Ministry of Justice on measuring product manager performance. So here’s an update to my original article from a real-life case study.

A bit of context #

Let me put this in context for you. When I wrote the original post about how to spot a good product manager, my experience up until that point had mostly been in the kind of companies that regarded product management as an evolution of project management – very process-driven. Gantt charts at dawn!

There, the product manager was meant to follow a waterfall process and attempt to make the user interface look like it hadn’t been designed by a database developer. (Even though it had.)

As a result my initial take on measuring how well a product manager is doing was against that backdrop.

Since then, I’ve learned a helluva lot from the dozens of different organisations I’ve worked with in the private and public sectors, many of which are embracing a more up-to-date approach to product delivery (i.e. user-centric, evidence-led, multi-disciplinary teams).

A terrible idea #

The Frost Report - Class Sketch (Credit BBC)

Credit: BBC

In the UK’s Ministry of Justice, I was Head of Product for their digital team, during which time I was responsible for 12 product managers of varying degrees of experience. Each product manager worked with her or his own multi-disciplinary delivery team (product management, research, design, development and delivery management).

Misguidedly, the senior management team was clinging on to their attempt to apply a command and control structure to a capable team of autonomous and intelligent people. Not the right approach at all. They had asked me to “stack rank” my product managers (assess their performance and put them in order from best to worst). There’s rarely any other reason for doing this other than a desire to reduce headcount by culling the worst performers.1

It didn’t appear to occur to them – though I pointed it out repeatedly – that this was frankly a terrible idea.

Hurray for diversity #

For starters, I knew that each person on my team was already a strong performer. Some of them had been in product management far longer that I had. Others were relatively new product managers, but even considering that were doing just fine, thanks.

Even if it was possible to assess all 12 product managers on a relative, linear scale normalised for their relative levels of experience, my team was perfectly capable of reading between the lines – they’d object so much to the stack rank exercise that they’d probably leave on principle. And I wouldn’t blame them for doing so, because who doesn’t love being devalued and treated as a commodity, right?

Each member of my team had different strengths and weaknesses, different backgrounds and levels of experience, different skill sets and different approaches. Sure, they were all product managers, but they each did product management differently. This was exactly what I needed them to do, because they were all solving different user problems.

Between compliance and disobedience #

It was becoming clear that I couldn’t refuse to assess their performance outright. This would lead to my swift removal, then someone less protective of my team than me would simply pick an arbitrary metric, rank my team and fire a few of them. I could serve my team better by treading a fine line between compliance and overt disobedience. I would do some kind of performance assessment, but I’d structure it so that a stack rank was bloody hard to do.

This is what I came up with.

Start with the team #

First of all, I sat down with my team and talked through what was going on. I then surveyed them to establish what they felt were the most important soft (emotional) and hard (technical) skills a product manager needed. We used the top ten or so as the dimensions for their assessment. I also asked them how conversant they were with skills from other disciplines they frequently would interact with, such as research, design, content and development.

Click on image to enlarge

Then I got them to assess themselves individually. They gave themselves a score out of 5 based on how skilled they felt they were in each area. I then plotted the results for each individual on a polar chart, and compared that to the team average.

Immediately it was clear to see where the team felt it was stronger and weaker. After seeing the results, several of my team took the initiative to start filling in the gaps: one person buddied up with another product manager who was more experienced in Kanban, another put themselves forward to do more public speaking, and so on.

Peer feedback #

But the really interesting part was yet to come. I asked each of the delivery teams to rate (anonymously) their product manager on the same sets of skills. A product manager needs to be effective in their interactions with their multi-disciplinary team, so I was interested in their team’s subjective perceptions of their product manager. Were they good communicators? Did they inspire?

Click on image to enlarge

This exercise again yielded some interesting insights. In some areas the product manager’s self-assessment and their delivery team’s assessment were in agreement. In others, the product manager had rated themselves weaker in an area, while the team thought they were particularly strong – and vice versa. These findings were also worth exploring further.

The distance between us #

Lastly, I asked senior management to rate the product managers in the same way I’d asked the delivery teams to. With a couple of notable exceptions, it was clear that senior managers had next to no idea how to answer the questions – they simply weren’t close enough to the product managers to make any kind of assessment.

While this highlighted a failing of the senior management team in what was declaring itself to be a ‘flat organisation’ (yeah, right), it also was a point for improvement in the product managers. Given how reliant they were on having senior management’s support to keep things moving apace, clearly they needed to invest more time into stakeholder management.

Takeaways #

So, what do I want you take away from this?

Measuring performance is multi-dimensional – a role requiring multiple skills means you need to assess multiple performance areas

There is no single, absolute measure of performance – people have different strengths and weaknesses (and that’s okay)

You don’t want a homogeneous team – user problems are not all identical so you want a mixture of personalities, experience and abilities on your team to tackle them

Never underestimate the power of a pretty graph – it’s never a waste to spend a bit more time on making your stats easier to consume and tell their story more easily

Examples #

Click the image for the live demo

I used Highcharts to render the charts. Each product manager had their own page – for their eyes only. Senior management only got to see the aggregated averages in case they got any big ideas.

I’ve got a live demo of the full report page available if you fancy taking a look. Needless to say, for this article the individual’s data has been anonymised (there was no product manager called Petra) and slightly randomised.

The two surveys that I used are here:

Similar versions of the self-assessment survey were also used for the product manager’s team and for senior management.

If you have any questions, pop them in the comments below.

  1. This was a few years ago, thankfully things are far more sensible on the Ministry of Justice Digital team these days.

Read more from Jock

The Practitioner's Guide to Product Management by Jock Busuttil

“This book has been a genuine game changer for me”
– pb (Amazon)

“I wish this book was published when I started out in product management”
– KejiA (Amazon)

8 thoughts on “62: How to measure product manager performance

  1. I would first start with what problems product managers are hired to solve for a company. Different companies hire product managers for different reasons, but typically there is a problem with making and executing product decisions. Example problems that product managers solve:

    1. Products don’t provide value to prospective buyers and users.
    2. Developers don’t know what to build, and why.
    3. Sales and marcom can’t consistently articulate the value of products.
    4. The process of learning the market is slow and unreliable.

    As you can see, the problems primarily relate to stakeholders on the team who are not empowered to build, market, and sell a product that delivers true value to customers.

    The question, then, is the extent to which a product manager is empowering the team to make sound product decisions.

    Accordingly, we turn to the stakeholders on the team as part of a “360 review” on whether the product manager is empowering them to be as effective as they can be. We ask questions such as:

    1. Do all the stakeholders understand the product’s unique value proposition?
    2. Do the stakeholders have confidence that product decisions reflect an understanding of the ever-changing market?
    3. Do the stakeholders believe the pace of learning is sufficiently fast?
    4. Do developers know why they are building what they are building?
    5. Does the product actually deliver its unique value proposition?

    I might add another leadership question:

    6. To what extent is the product manager helping each member of the apply her unique talents?

    To be sure, product managers need certain talents and skills to perform well. But the performance of a product manager – the extent to which those talents and skills are addressing the challenges that product teams face – lies in the answers to these sorts of questions.

    • Thank you for taking the time to comment. The points you raise certainly reflect what happens in real life, but I think they also highlight the problems I’ve seen with the way product manager performance is typically done.

      My main issue is that product managers are often assessed on:

      1. the performance of their products; or

      2. the performance of factors outside their direct control, such as the performance of other teams.

      A product may be performing well or badly for a number of different reasons, but it is perfectly possible for a well-performing product to be managed badly by a product manager, and vice versa. So product performance by itself isn’t a reliable indicator for how well the product manager is doing.

      Similarly, teams such as Marketing and Sales should be expected to do their own jobs well, but again whether they do or not is often independent of whether the product manager is doing her job well.

      I expand on this in the original performance article I wrote.

      Lastly, you put a lot of emphasis on alignment across internal stakeholders and the delivery team. I do agree that these are important parts of a product manager’s role – it would be very difficult to create a successful product without good buy-in and alignment. Again by itself it is not the only aspect of a product manager’s performance to take into account.

      There’s also the reality that not every internal stakeholder is necessarily in favour of the product (or the product manager), perhaps because the product threatens their fiefdom or competes with their own plans. In these situations the product manager’s stakeholder strategy might be to minimise the negative influence of ‘anti’ stakeholders and concentrate on the ‘pro’ and undecided stakeholders. But this is a complex and lengthy discussion for another article, I think :-)

      • Jock, note that, contrary to your assertion, I actually did not propose any evaluation criteria that assess product performance. Rather, I enumerated the problems a product manager solves, correlating to functions that a product manager performs to address these problems that stand in the way of product success.

        I then enumerated questions that shed light on the extent to which the product manager is successfully performing these functions. I did not mention product sales, revenue, or any other product performance metric.

        As you contend, many factors outside a product manager’s direct purview contribute to the performance of a product. If developers don’t competently carry out their duties, product performance will suffer. If sales doesn’t competently perform its functions, product performance will suffer. If marketing doesn’t competently perform its functions, product performance will suffer. I blogged about this issue in 2005, writing then that I did not believe product performance equals product management performance.

        But to the extent a product manager hasn’t facilitated process that empowers, with strategy and context, developers, sales, and marketing to succeed, she is responsible.

        You can make the case that the answers to the evaluative questions I enumerated still depend on the competence and abilities of others. I welcome tweaking the questions to get more directly at the product manager contribution.
        .

      • Thanks for the further comments, I appreciate you taking the time to explain your points.

        I may have misunderstood the points you were trying to make, but what I thought you were saying was that product managers should be assessed on their ability to solve the problems they were hired to solve. You then provided a few examples – let me illustrate why I think they can sometimes pose a problem for assessing product manager performance:

        1. Products don’t provide value to prospective buyers and users.

        In this situation, the product in its current state is fundamentally flawed. This could be for a variety reasons, perhaps user research didn’t happen before product build, so the wrong user problem (or no user problem) is being solved by the product; perhaps the market has evolved since the product was built and now there is a significantly different problem to solve; or perhaps the product build was executed poorly and so failed to solve the problem it set out to in the first place.

        Regardless of the cause, I would expect a good product manager to redo the user research with their team, then figure out whether it’s still a valuable enough problem to solve, whether it’s technically possible to solve the user problem, and whether it’s still financially viable to create a production-class replacement.

        This is likely to mean a significant reset of the product, perhaps even discarding the current product completely to avoid pouring good money after bad.

        When thinking about product manager performance, all too often I see the situation where a product manager is seen to be underperforming because they’ve decided to kill a flawed product – the organisation only sees the investment they’ve already made in the product so far and can’t bear to throw it away, despite it being a sunk cost already.

        2. Developers don’t know what to build, and why.

        Developers not knowing what to build would suggest they’re waiting for someone to spoon-feed them requirements. I don’t believe this is a good way for delivery teams to operate. What the team needs to be working on is determined by the team in response to their evolving understanding of the user problem they’re trying to solve, and later how well their product is solving the user problem.

        Now, if the developers don’t understand the user problem they’re trying to solve in the first place, and why it matters, then this points to a wider dysfunction, namely that they as a team are not close enough to their users. While this is certainly something a product manager can help to solve, I’d argue it’s by no means her problem to solve solo.

        The whole team – developers, designers, user researchers, content designers, delivery/project managers and product managers – should all get at least a couple of hours in front of real users every six weeks.

        In other words, developers and everyone else on the team should have intimate understanding of user problems because they’re all taking the responsibility to go out and observe them, rather than waiting for the product manager to proxy it to them.

        3. Sales and marcom can’t consistently articulate the value of products.

        Call me a cynic, but I’ve never seen any correlation between the quality of information provided by a product manager in traditional company setups to the Sales and Marcom teams, and the ability of Sales and Marcom to articulate product value. I’m never writing another “2-pager” for anyone ever again :-)

        Similar to the point above, it’s the responsibility of sales and marcom people to get closer to the user research that a delivery team is generating, just as much as it’s the delivery team’s responsibility to make that user research as visible and easy to understand as possible.

        Good examples I’ve seen include delivery teams giving all-access “show the thing” talks every week. Key findings from user research and discovery are shared with whoever in the organisation wants to turn up and listen. Other examples include posting up recent research findings in public areas like a break room or cafeteria.

        Again, a product manager can help to solve this, but is by no means the sole provider of the solution.

        4. The process of learning the market is slow and unreliable.

        I’m not 100% sure whether you mean the speed with which a new product manager learns about the market for their product, or whether you mean how well the organisation as a whole conducts its market research.

        If the former, then I agree, a product manager should be able to get up to speed with a new product and market quickly – my view is they should be in a position to make informed decisions about their product within a month of joining.

        If the latter, then again it’s a broader dysfunction in the organisations and not one a product manager can solve solo. It would suggest to me that market research is too broad and unfocused to be meaningful, or that market research is too far removed from the user research into specific problems that products could be solving.

        Generally, the problems I see in specific delivery teams, especially startups, is a desire to try and solve to big a market problem for too large a group of users. There are too many nuances and exceptions because the problem by its nature is complex and multifaceted.

        Instead, with limited time (or funding runway) and people available to successfully and viably solve a market problem, my advice is always to narrow the focus, to find a smaller problem to solve for a more specific group of users. Then look at adjacent problems for the same set of users, or adjacent user groups for similar problems, and expand from there.

        Hopefully that gives you a bit more context on why I don’t think the ability of a product manager to solve these problems you’ve suggested are necessarily the best way to assess their performance.

        I’m also not saying that the way I happen to have done it at MOJ is the only way to do it – I just want product managers to be assessed primarily on their ability to be a product manager, not their ability to perform other roles outside of product management.

        I totally concede this is hard to delineate, given that product managers often end up fill gaps on teams that would typically be occupied by user researchers, designers and so on, but that’s the point of discussing it.

        Thanks again!

      • Jock, I’m going to take each of your points individually and then propose a simple, overarching clarification that may help us reach agreement.

        When thinking about product manager performance, all too often I see the situation where a product manager is seen to be underperforming because they’ve decided to kill a flawed product – the organisation only sees the investment they’ve already made in the product so far and can’t bear to throw it away, despite it being a sunk cost already.

        Yes. However, this point argues in favor of looking at the product manager’s performance through the lens of building and selling products that provide value. A product manager who inherits a product and decides to throw it away is not responsible for the company having attempted to sell a product that provides little to no value. On the contrary – she has solved that problem.

        Developers not knowing what to build would suggest they’re waiting for someone to spoon-feed them requirements. I don’t believe this is a good way for delivery teams to operate. What the team needs to be working on is determined by the team in response to their evolving understanding of the user problem they’re trying to solve, and later how well their product is solving the user problem.

        The requirement is the problem, or the absence of it, as I have argued since 2005. Developers not knowing what to build comes precisely from not understanding the market problems and from not having a shared understanding of which ones the team has chosen to solve.

        It is true that product managers do not bear 100% responsibility for ensuring each and every developer understands these things, but product managers lead that process, including by encouraging and facilitating access to customers.

        Call me a cynic, but I’ve never seen any correlation between the quality of information provided by a product manager in traditional company setups to the Sales and Marcom teams, and the ability of Sales and Marcom to articulate product value. I’m never writing another “2-pager” for anyone ever again :-)

        Who said anything about a product manager throwing information at sales and marketing?

        The most effective product managers facilitate conversations, leading to a shared understanding of the target market, that empowers sales and marketing to articulate the value of the product in terms of market problems.

        I’m not 100% sure whether you mean the speed with which a new product manager learns about the market for their product, or whether you mean how well the organisation as a whole conducts its market research.

        I meant how quickly the team learns. To learn quickly, the team needs leadership and processes that facilitate learning. Some of the most effective product managers know how to work with, and coach, teams to design experiments. And as I mentioned previously, facilitating conversations about, and access to, customers can also accelerate learning about the market.

        So insofar as they are successful in applying this sort of leadership, product managers empower teams to learn faster.

        The Bottom Line

        We can evaluate product managers in terms of their abilities or in terms of their contribution to outcomes that provide value to the organization. Your proposed evaluation criteria focus on abilities and not on actual contribution to valuable outcomes.

        You have rightly pointed out that a product manager cannot single-handedly solve the problems she is hired to solve. Thus we cannot simply evaluate her performance in terms of having solved these problems.

        However, note that I have repeatedly used such words as “empowerment”, “contribution”, “facilitation”, and “extent”. We can and should measure the contributions a product manager makes to solving the problems. Measuring such contributions will not be scientific or rigorous, but we shouldn’t give up and just measure her abilities.

        A 360 review of internal stakeholders, combined with the right questions that get at the extent to which the product manager, with her limited role, empowers the team to overcome the problems, can be a useful indicator of her performance.

      • I think I understand where we differ in opinion.

        Measuring product managers on their contribution to outcomes that the organisation values only works when the organisation values the right things – and that’s not always the case.

        In an ideal scenario, an organisation will recognise how good product management actions ultimately deliver value to the organisation. In this case, I absolutely agree that a product manager’s performance can be measured on their contribution towards good outcomes.

        However this relies on the organisation having a good understanding of modern product management, which is still not widespread.

        More often than not, organisations tend to place value more on macro measures such as revenue / profit, market share, reduced cost base, customer satisfaction and similar.

        In larger organisations, a product manager typically can indirectly influence these factors, but is not often in direct control of them. In smaller organisations, there is more scope for a product manager to affect them directly.

        And I think that’s why we differ in opinion. Your experiences may well differ to mine, and that’s okay. I don’t think this is a “right way” or “wrong way” kind of discussion. We’ve both offered different approaches, and it’s up to readers of this blog to figure out which of the many available approaches suits their situation best.

        Thanks again for contributing to the debate!

      • Jock, we agree that not all companies understand product management or agree on the value good product management provides.

        But the primary topic of this discussion has been about how companies should assess the performance of product managers, not about how they do assess it or whether they agree on what value a well-performing product manager provides.

        As every good product manager knows, value is rooted in solving problems. Accordingly, I enumerated the problems a product manager provides leadership in solving. I then went on to suggest questions that shed light on the extent of a product manager’s contributions to solving the problems.

        In summary, the approach I have suggested measures, and provides insights into, the impact the product manager’s application of her talents and skills is having on solving problems, and therefore providing (real, not perceived) value to the organization.

        There is nothing wrong with assessing a product manager’s skills and abilities. But skills and abilities are not the same as performance. Moreover, as I have argued, measures of product manager performance, as I have defined them, are not the same as measures of product performance, and they embrace that factors outside of a product manager’s control affect ultimate outcomes.

      • Respectfully, I disagree that “skills and abilities are not the same as performance” – I believe a product manager’s performance encompasses both their skills and abilities, as well as their effectiveness, but I’ve already covered these points.

Comments are closed.