How not to Create a Culture of Experimentation

It is the job of a Product Leader to ensure there is a healthy product culture within their organization.  A key part of this healthy culture is one of ensuring that teams feel safe to raise ideas and try things.  After all, we know that developing new products is a high risk activity — Will anyone pay for this? Can we build it?  Will they like it?  Can we sustain a business around it?

These questions are typically not answered with 100% confidence.  More than likely, there is a great deal of risk in each of the answers.  Companies fail to address these risks and cease to innovate when they ignore these risks and assume they can guess the way to a certain future.

Modern product thinking recognizes that a repeatable way to address risks is to use the scientific method of defining a hypothesis and run one or more experiments to validate/invalidate it. 

Split Testing Experiment
Example of Experimentation

Suppose your product analytics show potential users dropping off during on-boarding to your app right at the sign-up form.  Based on some anecdotal evidence you believe that the reason you lose potential users here is because sign-up has so much friction.  Further, you hypothesize that adding Google Integrated Sign-In would reduce this leaky funnel.

Rather than building the full integration, you can run a quick experiment.  Simply, add the option for Google SSO as a sign-up option in a fraction of the time. When the user clicks this option you will evidence to support your hypothesis. To have even better data, you direct them to a message saying this option is not yet available – would they like to sign-up using an alternative method?

If the user chooses an alternative method you just learned this was not the problem but Google SSO is preferable.   If they choose “No Thanks” and exit the process there you have strong signal confirming your hypothesis.  You just reduced the risk of developing a useful product feature.

Another way we reduce our exposure to risk is through agile and lean product developments.  Specifically, the idea of slicing large projects into smaller increments of value delivery.  Like Experiments, this gives us a huge benefit of learning and adjusting before investing a more significant amount into capability.

In this article I want to give a very specific example of a mistake I made as as relatively new Director of Product Management.

Move Fast and Break Things

Mark Zuckerberg famously created an internal development motto for its developers “move fast and break things”. They have since made this a more measured motto involving stable infrastructure reflecting the side unintended side effects.

While many in the industry (and beyond) now pan this mentality, it drove a culture of trying things. When your top leader, gives you the space to be comfortable in putting your idea into action, you know that failure is ok. This reflected the understanding that if you want a workforce to innovate in the face of risk – failure will occur.

The healthy version of this today, is that organizations want to “Learn Fast”. This positive spin reflects a supporting perspective to the individual and team, plus the real desire to create fast feedback loops as iterate toward increasing product goals.

My Failing in Creating a Healthy Product Culture

Earlier in my career, I have a small team of product managers working for me.  I was being slowly self-taught as a product leader and just figuring out how to manage a team of people that were mostly new to the product management discipline.  We had a lot of professional development to do.

Still, the whole team was awesome and wanted to push things forward.

One of the PMs on my team had an idea he was excited about.  It may have come from client conversations or simply pattern matching with other technology products at the time.  Our workflow product had a note taking component.  Some users would record a lot of notes as they progressed items in the workflow and we wanted to enable them a quick way to find all notes referencing key data like accounts and customer names.  This would eventually save them time in audit processes.

The idea was to leverage a popular feature found in online discussion forums at the time, tagging.  I pushed back on whether this was the right solution for the problem but it was his product.  He worked with the team to design the feature and even demonstrated a mockup at a user group prior to release.

When the updated product was released with his feature (this was on-premises software), it never used.  He went around teaching lead users about the feature and its benefits but they just never adopted it.

He learned a lot from this experience about validating his idea more before investing in it.

Unfortunately I think I taught him something else too

For months after the release, I would ask him how the feature was doing.  Do we know if anyone is using it yet?  Do we have any support requests on it?  Does sales use it in demos?

The answers became consistently clear – the feature either didn’t solve a real problem or was not a good solution for it.

As a flawed human I did not understand the impact of the continued questions when I knew the answer.  To me it become a joke when he would be presenting another planned feature.  Worse, I might joke about it in the company of other team members.

I was inhibiting a healthy product culture to grow.  Who would want to propose innovative ideas if this is the possible outcome? 

I slowly came to this realization as I saw his work, in particular, get more conservative.  With the long cycles we had of getting new releases into wide production use, it was nearly a year before I asked him about this issue and I resolved to fix it.

My lesson here, especially for newer product leaders is not to be like this.  Instead, pick your team up when something they try does not works as expected.  Use the language of learning rather than failure.  

Coach them:

  • Learning should be celebrated.
  • That this is exactly what you want.  Learn and adjust.
  • Help them discover ways to learn quicker whenever possible.



This story came to my mind the other day and I couldn’t help but hope that I am a better product leader today.  We have much better practices for fast learning that are widely known and used.   Yet, still, it is informative that you may think you follow all the better practices available today but your approach to working with the team is a critical element of a healthy product culture.

If you want to drive innovative you must nurture the right culture.