Expensive and risky assumptions (and why you should check them)

I gave a talk recently about how I’ve been using data and analytics to guide my decisions in product management. I’ve edited the transcript a little and split it into bite-size parts for your entertainment. This bit is about expensive and risky assumptions (and why you should check them).

Data from the outset #

Because I think it may be of some help to you in putting through our recommendation for additional men for the Promotion Department, I am outlining briefly below the duties and responsibilities of the brand men.

This slightly blurry typescript is from 1931. It’s a memo that a chap called Neil McElroy wrote to his bosses. He used to work at Procter and Gamble – P&G. He was responsible for the soap product Camay (which still exists today). He was also really frustrated because his product was always playing second fiddle to another soap brand at P&G called Ivory.

So he decided that the best way to get more of an advantage with his product was to propose the creation of a new kind of role, what he called a “brand manager”. The reason I mention this – bearing in mind this is back in 1931 – is because this is pretty much a prototype for what we call “product managers” today. What he proposed was a role that would work and coordinate with other departments and, more importantly, he advocated a data-driven approach.

(e) Keep whatever records are necessary, and make whatever field studies are necessary to determine whether the plan has produced the expected results.

As McElroy says, the brand manager needed “to make whatever field studies are necessary to determine whether the plan [for the product or brand] has produced the expected results.”

So even at the very beginning, product management relied on data and evidence.

The product manager does not specify the product #

Now there are plenty of examples of failed products that illustrate the value of having data and evidence.

We still get the situation where some companies are expecting their product managers to come up with the ‘requirements’ for their product – even though they’ve never spoken to the actual people who will be using the product.

Then there’s the other aspect that, unless you’re building a software product that will be used by product managers, the product manager isn’t representative of the users either. So really not the right person to be determining what the product should be in the absence of any evidence.

There’s so much written on this topic already that it would be churlish for me to reproduce it all here. To cut a potentially long story short: a product manager and their team must get out there and meet the actual users of the product – no exceptions.

If you need more on that, there’s plenty from Marty Cagan, who wrote a book called Imagine and blogs regularly on SVPG, and also Steve Blank, who has dozens of great videos on his site and also The Four Steps To The Epiphany book. They and many others make the case for why it’s so important for product managers and their teams to get out of the building to actually meet their users and gather some evidence.

Assumptions and risks #

So there are plenty of examples of  products that really illustrate the value of having data and evidence. One particular failure I quite like using is this one.

Back in 2001, right in the middle of the dot-com bubble1, there was a brilliant engineer called Dean Kamen.

By then, he’d already had a decent string of successes under his belt, including an innovative and successful insulin pump, and a motorised wheelchair that could, amongst other things, go up and down stairs, which was amazing, but also allowed the occupant to raise themselves to standing height so they could see people eye-to-eye.

Now, unlike his previous successes, his next project was much more ambitious. This time, in his own words, he wanted to reinvent personal transportation for everyone. He said it was going to be as big a leap forward from the car as the car had been from the horse and buggy.

Before he’d even launched this world-changing product, the patent application got leaked, and this sent the dotcom world into a frenzy of speculation about this thing called Project Ginger. And they knew it was about transport, but they didn’t know what it was. So some people were thinking it was something crazy like a jet-propelled scooter. And some other guys were thinking maybe it was a Star Trek transporter or something. But in reality it was actually much more down-to-earth:

Yes, Dean Kamen invented the Segway, which – if you’re not familiar with it – is a self-balancing, two-wheeled electric scooter. So not quite the Star Trek style of transporter people were expecting.

Now, the thing is, whilst it’s a bit of an anticlimax, actually as a piece of engineering, it was pretty marvellous – it did its job very well. If you think about what processors and computing power were back then, it did the job of being a self-balancing, electric scooter very well. So, given all of that, and given his track record, and given the actual build of the product was great, why was it not the world-changer he’d expected it to be? Why are we all not riding our Segways today?

Of course, no product could have lived up to the amount of hype the Segway received before it was launched. So let’s look at the assumptions that Dean and his team had made before they built the thing.

Product: top speed 12kph

Process: time between charges

Market: everyone will want one

Pricing: $4,995

User: won’t feel stupid on one

Regulatory: legal to ride

They made assumptions about:

  • the product – were they the right features?
  • how the product would be used – would it be convenient?
  • the market demand and the price people would be happy to pay for it
  • how the users of the product would feel about it; and
  • most crucially, they made assumptions about the regulations that surrounded the product.

When they launched the Segway, it was illegal to ride in 32 US states and the District of Columbia. That’s quite a sizeable barrier to market.

And with over $100 million US dollars already invested in the Segway, the company had to spend even more money to lobby each state to change its laws to allow the Segway to be used.

Now the reason I mention all that is: don’t you think it would have been much better to check – particularly this last point about whether it was legal to ride – before they’d got to the expensive business of mass producing it?

Assumptions = risks #

So really, the first takeaway I have for you is that your assumptions – the things you think you know about your product, about your users, all these assumptions you’re making without any evidence – translate directly into risks for your product.

Next time: why we can’t help jumping to conclusions.

  1. Okay, in my own book, I actually have a footnote which says “March 10, 2000, is considered the date on which the dot-​com bubble burst”, so best go with that.

Agree? Disagree? Join the conversation: