Mastering Experimentation: How to Drive Innovation and Insight in Your Team

Introduction

A few weeks ago, I shared a LinkedIn photo of a team experimentation session I conducted, focusing on team behaviours and work patterns. It received a fair bit of love, so I figured why not share a few posts about my approach to experimentation? Today’s email is about getting us all on the same page about the term “experimentation” so we have a common language.

With my current contract, I am partnering with the organization to drive internal innovation and experimentation in EX. From a strategy point of view, I am looking at the business’s internal portfolio of EX products, services, experiences, functions, and teams and analyzing these for:

• Driving pains

• Where the biggest gains and opportunities are

• Which should fall into our incubator

• Which needs to be sunset

The other part of my role is designing and scaling out an org-wide innovation framework, measurement engine, team practices, and experimentation. So I figured, why not zoom in on experimentation, given it received some love?

Before I get into this, it is important to note that having the right behaviours and mindset is vital to having an anti-fragile team. If you are new here, then read this first

What is experimentation?

Experimentation is something we do every day. If you want to observe experimentation in the wild, spend some time with young children or toddlers. They are constantly in sense-making mode and experimenting, usually by putting things in their mouths.

Before we get into team experimentation, let’s break down and clarify what experimentation means. For some, the word is welcoming and exciting; for others, it can be daunting and non-inclusive. For me, experimentation is:

A method to validate or reject a set of hypotheses and assumptions, or to discover something new.

Let’s assume we all agree on the above meaning of experimentation. We can quickly look at where it adds value.

The value of experimentation

Learning: This is ultimately the end goal of every experiment. It’s not about whether an experiment works or doesn’t work. What we are trying to do is gain richer insight and understanding into why and how it worked or didn’t work. One of the traps in experimentation is when the experimenter wants the experiment to be successful. This bias can show up in many ways (more on that later).

Early solution signals: We can sometimes hit a challenge where we don’t have a potential solution to test. We can use experiments to test different variables of the current challenge to help identify potential opportunities. I do online coaching with a personal trainer. I gave them some feedback about their app being slow and not logging my workout. They couldn’t see the challenge on their side, so they ran a few experiments to see what might be causing it. This ranged from reducing the images on my version of the app to moving me to a different server and also running a code analyzer to help identify the challenge.

Discovery & Innovation: It goes without saying, but experimentation is vital to new breakthrough discoveries and innovation. To see hundreds of examples of this, just look into the past.

For example, the discovery of penicillin in 1928 by Sir Alexander Fleming. As the story goes, he was coming back from his holiday and noticed that a Staphylococcus bacteria in a petri dish had a mould growing on it called Penicillium notatum. This penicillium was effective in stopping the staphylococcus from growing. It was the penicillium’s self-defence mechanism that was effective in killing the staphylococcus. This was massive for medicine and led to the development of the antibiotic.

Behaviour Change: We use experimentation here to create a safe space for managers, teams, or individuals to test out new behaviours or ways of working. We map this to a tension felt in the team, and other factors like internal influence, attitudes, and perception. We then co-design an experiment with a new behaviour, activity, or approach and run it for a set time. This is less about boring change frameworks that often don’t work in the real world and more about taking a hands-on approach to testing new behaviour or ways of working.

Testing: Often, the main use of experimentation is to test something, usually a hypothesis about how or what will happen. Here, they will run the experiment to test if the hypotheses were validated and if the outcomes aligned.

Ongoing development: Often seen in big platforms like Facebook, Amazon, and so on. These platforms are running a lot of little experiments like A/B testing to constantly develop their platforms. In 2011, Amazon set up its internal experimentation platform and openly talked about experimentation in an old letter to shareholders. Imagine how many they run now in 2023.

Sounds exciting, right? And pretty simple… slow down, my friend. We know what the word means, and we know where it can add value. Let’s look at some of the booby traps that can sneak in and ruin the whole thing.

Booby Traps & Watch Outs

Bias: We touched on earlier on how bias can sneak in, but bias can be seen in several ways. Here are a few to get you going:

• Confirmation bias: Where we tend to find, interpret, and often favour what confirms our assumption or belief.

• Sampling bias: Often where we don’t bring in a fair spread across the spectrum, this could be people, backgrounds, etc. This drives an unfair representation or conclusion.

• Performance bias: Often where an experiment may get more special treatment than the others because it is seen to be cooler or different, etc.

Variables: After bias, we have variables to think about. There will always be a selection of variables in every experiment (I like to think of these as dials on a machine). These variables are summed up in three key groups:

• Independent: These tend to be variables in the experiment that are manipulated, tweaked, or changed throughout the run time of the experiment.

• Dependent: These are the variables that often get measured or observed, depending on the change made in the independent variable.

• Control: These variables are kept consistent to ensure that any changes observed are due only to the changing of the independent variables.

As a dyslexic, when a word is used in various manners too close to each other, my brain turns into knotty spaghetti. So here is a high-level example to bring it to life. Let’s assume your experiment is around how the length of your working day affects how productive you are:

• An independent variable (the element we are changing) would be the number of hours in a working day. Let’s assume you go to 5 hours, while another group remains at 8 hours.

• The dependent variable (the thing we are measuring or observing) would be the quality or quantity of work completed in the 5-hour workday or something similar.

• The controlled variable (the element that stays consistent) would be the space where they work, the work type, the amount of one-on-one time with managers, etc.

Of course, there are many other factors to the simplified example, but you see how X doesn’t always equal Y when you start to factor in variables. One more thing to note that gets overlooked is the following:

Controls: When you run an experiment, you always want to have a controlled group. This is a group/segmentation that does not know about or go through the experiment. This helps later on down the line when you start to compare and distil the effects of the variables.

Here’s the rub:

Experimentation is fun, it’s always insightful, and it nearly always leaves you with “aha” moments. While the word may feel daunting, it’s nothing to shy away from. If children and toddlers can do it, I’m sure you can too. The only real difference is that before running any internal experiments, you have to be mindful of:

• Bias

• Variables

• Controls