Contributed by Sierrah Coleman.
Sierrah is a Senior Product Manager with expertise in AI/ML, predictive AI, and recommendation systems. She has led cross-functional teams at companies like Indeed, Cisco, and now Angi, where she developed and launched scalable, data-driven products that enhanced user engagement and business growth. Sierrah specialises in optimising recommendation relevance, driving AI-powered solutions, and implementing agile practices.
In product management, people often say: “fail fast,” “fail forward,” and “fail better.” But the reality is that failure isn’t valuable unless you learn something meaningful from it.
Product experiments are often viewed through a binary lens: Did the test win or lose? This yes-or-no framing may work for go/no-go decisions, but it’s an ineffective approach to driving real progress. The most powerful experiments aren’t verdicts—they’re diagnostics. They expose hidden dynamics, challenge assumptions, and reveal new opportunities for your platform. To build more innovative products, we must design experiments that teach, not just decide.
Learning > Winning
Winning an experiment feels rewarding. It validates the team’s work and is often seen as a sign of success. However, important questions may remain: What exactly made it successful?
Conversely, a “losing” test is sometimes dismissed without extracting insight from the failure—a missed opportunity. Whether a test “wins” or “loses,” its purpose should be to deepen the team’s understanding of users, systems, and the mechanics of change.
Therefore, a strong experimentation culture prioritizes learning over winning. Teams grounded in this mindset ask: What will this experiment teach us, regardless of the result?
When teams focus on learning, they uncover product insights on a deeper level. For example, suppose a new feature meant to increase engagement fails. To understand the underlying issue, a dedicated team might analyze user feedback, session recordings, and drop-off points. In doing so, each experiment becomes a stepping stone for progress.
Experiments also foster curiosity and resilience. Team members become more comfortable with uncertainty, feel encouraged to try unconventional ideas, and embrace unexpected outcomes. This mindset reframes failure as a source of knowledge—not a setback.
How to Design Tests That Teach
To make experimentation worthwhile, you need frameworks that move beyond binary outcomes. Well-designed experiments should explain why something worked—or why it didn’t. Below are three frameworks I’ve used successfully:
- Pre-mortems: Assume Failure, Learn Early
Before launching a test, pause and imagine it fails. Then ask: Why? This pre-mortem approach reveals hidden assumptions, uncovers design flaws, and helps clarify your learning goals. Why are you really running this experiment?
By predicting failure scenarios, teams can better define success criteria and prepare backup hypotheses in advance.
Pre-mortems are especially useful when diverse perspectives are involved. For example, designers, product managers, and customer support specialists may surface unique risks and blind spots that a single-function team could miss.
- Counterfactual Thinking
Instead of asking, “Did the experiment win or lose?”, ask: “What would have happened if we hadn’t made this change?” This mindset—known as counterfactual thinking—encourages deeper analysis.
When paired with historical data or simulations, teams can “replay” user interactions under different conditions to isolate the impact of a specific change. This approach not only identifies whether something worked—it reveals how and why it worked.
Counterfactual analysis also helps teams avoid false positives. By comparing actual results against initial hypotheses, they can separate the true effect of a change from external factors like seasonality, market shifts, or concurrent product releases. The result? More accurate experimental conclusions.
- Offline Simulations
When live testing is slow, expensive, or risky—simulate instead. Offline simulations allow you to control variables, model edge cases, and iterate quickly without exposing real users to unproven changes.
Simulations improve precision by offering detailed environment breakdowns, isolating variables, and uncovering scenarios that live tests might miss. They also create a low-risk space for new team members to explore ideas and build confidence through iteration.
Case Study: Building an Offline Simulator to Learn Faster, Not Just Fail Faster
At Indeed, our recommender systems powered job search experiences by ranking results, suggesting jobs, and personalizing interactions. Improving these models was a priority. However, the process was slow—each change required a live A/B test, which meant long timelines, engineering overhead, and user risk.
This limited the number of experiments we could run and delayed learning when things didn’t work. We needed a better path forward.
The Solution: Build an Offline Simulator
I partnered with our data science team to build an offline simulation platform. The idea was simple: What if we could test recommendation models without real users?
Together, we applied the three strategies above:
- Pre-mortem mindset: We assumed some models would underperform and defined the insights we needed from those failures.
- Synthetic user journeys: We modeled realistic and edge-case behaviors using synthetic data to simulate diverse search patterns.
- Counterfactual analysis: We replayed past user data through proposed models to evaluate performance under the same conditions, uncovering hidden trade-offs before deployment.
This approach didn’t just predict whether a model would win—it helped explain why by breaking down performance across cohorts, queries, and interaction types.
The Impact
The simulation platform became a key pre-evaluation tool. It helped us:
- Reduce reliance on risky live tests in early stages
- Discard underperforming model candidates before they reached production
- Cut iteration timelines by 33%, accelerating improvement cycles
- Design cleaner, more purpose-driven experiments
It shifted our mindset from “Did it work?” to “Why did it—or didn’t it—work?”
Culture Shift: From Testing to Teaching
If your experimentation culture revolves around shipping winners, you’re missing half the value. A true experiment should also educate. When every test becomes a learning opportunity, the return on experimentation multiplies.
So ask yourself: Is your next experiment designed to win, or designed to teach? If the answer is “to win,” then refocus it—because it should also teach.
Let your frameworks reveal more than just outcomes—let them reveal opportunities.
Finally, remember: designing tests that teach is a skill. It gets stronger with practice. Encourage teams to reflect on their hypotheses, iterate on setups, and keep refining their methods. The more you focus on learning, the more valuable your product insights will be.
Over time, your team will be better equipped to tackle complex challenges with confidence, curiosity, and creativity.
