Experimental Mindset

In previous articles, we covered the philosophy of Systems Thinking, both broadening and narrowing our aperture to understand how to look at feedback loops and boundaries. But how do we apply this to our job as a Solutions Architect? We believe the answer lies in the combination of knowing where to define the boundaries and embracing an experimental mindset.

Why does this matter? (“so what”) As an AWS Solutions Architect, you are already used to navigating complexity. But with each passing year, the software (and cloud, et al) industry continues to evolve at a rapid pace. So how do you adapt to the constant change? Think: Am I too reliant on what’s worked before? Do I still challenge my biases enough? Am I still working with a Day 1 mentality? These are questions that an experimental mindset - one that thrives on exploration, learning, and adaptation - can help with.

As evidence of rapid change:

“The democratization of technology has fundamentally changed the way businesses operate.”

Andy Jassy spoke these words at the AWS Summit in New York in 2019, and in the time since, AWS has expanded by 8 regions, and 34 full product launches.

Why Embrace an Experimental Mindset?

We live in a world full of ambiguity, and we are often tasked with solving problems where the best solution is not immediately clear. Even with all the data at our fingertips, there are no guarantees. But instead of seeing this uncertainty as an obstacle, you could (and should) treat it as an opportunity to experiment. The beauty of experiments is that they allow you to make progress through constant iteration, uncovering better ways to achieve your goals.

Jeff Bezos nailed it when he said, “It’s not an experiment if you know it’s going to work.” If you’re only pursuing projects where success is guaranteed, you’re not experimenting—you’re stagnating. A true experimental mindset pushes you to try new approaches, even if the outcome is uncertain. The more experiments you run, the faster you’ll discover what works.

“Invention requires two things: One, the ability to try a lot of experiments, and two, not having to live with the collateral damage of failed experiments.” - Andy Jassy

The Power of Reversible Decisions

At Amazon, we often talk about one-way and two-way doors, which really reflects a sense of how permanent a decision is. One-way doors are those choices you can’t easily undo—once you’ve made the call, you’re stuck with the outcome. These decisions require more caution and careful planning because the stakes are high. Two-way doors are different: you can make the decision, test it out, and if it doesn’t work, you can reverse it without much trouble.

Experiments thrive in a two-way door space. They allow us to move quickly, try something new, and pivot as needed without significant risk. Learning happens faster when you’re not stuck in decision paralysis.

Here’s where it gets interesting: many decisions that initially seem like one-way doors can actually be converted into two-way doors if approached with an experimental mindset. By breaking a big and irreversible decision into smaller and more testable steps, you can turn what feels like a high-stakes move into a series of low-risk experiments. For instance, instead of deploying a major system change all at once, you can run a pilot, gather feedback, and iterate. This approach not only reduces risk but also enhances learning, allowing you to gather data and adapt before fully committing.

The ability to convert one-way doors into two-way doors is a hallmark of a strong experimental mindset. It encourages you to test assumptions in a controlled way, allowing for more agile, informed decision-making. The more decisions you can turn into experiments, the more quickly you can innovate and evolve.

Learn and Be Curious

At the core of an experimental mindset is curiosity. The Learn and Be Curious LP encourages individuals to challenge their assumptions, explore new ideas, and never stop asking questions. Curiosity fuels experimentation. Without it, you’re just going through the motions, repeating the same actions and expecting different results. (Spoiler: that’s called insanity!)

“Learn and Be Curious: Leaders are never done learning and always seek to improve themselves. They are curious about new possibilities and act to explore them.”

This curiosity extends beyond work. Ever tried a productivity hack or dipped your toes into GTD (Getting Things Done)? These life experiments are often the first steps in adopting an experimental mindset. But while most people stop at these small personal improvements, SAs can and should take it further - applying that same curiosity to larger and business-critical challenges.

How to Set Up an Experiment

The process of setting up an experiment closely mirrors the Scientific Method, a structured approach that has been used for centuries to explore questions and solve problems. Like the Scientific Method, an experiment starts with defining a problem or formulating a question. You then develop a hypothesis (effective an educated guess) about what the outcome will be. From there, you test your hypothesis by running controlled experiments, gathering data, and measuring results against your expectations. Just as in science, if the data does not support your hypothesis, you adjust your approach and run a new batch of tests. The goal isn’t just to validate your initial assumptions but to learn through iteration, continuously refining your solution based on the evidence you collect. By framing your experiments this way, you ensure your process is methodical, data-driven, and repeatable - key elements of both the Scientific Method and an experimental mindset.

Get started using the following steps

1. Define the Problem

You can’t experiment if you don’t know what you’re experimenting on. Start by clearly defining the problem or opportunity. What exactly are you trying to solve? Why? and, so what does it matter? Too many experiments fail because they aren’t aligned with broader business objectives. Make sure the problem you’re tackling is relevant and worth solving. Use frameworks like the 5-Whys to help you get clarity.

2. Formulate a Hypothesis

Every experiment needs a clear hypothesis—a statement of what you believe will happen based on your knowledge and experience. Use the working backward framework, where you start with the desired outcome and then outline the steps needed to reach it. This ensures your experiment is goal-oriented, not just a random attempt at innovation.

3. Set the Metrics

Identify input goals—the actions and activities you control—and tie them to output goals, the results you hope to achieve.

4. Timebox and Iterate

An experiment without limits is just a project that never ends. Timebox your experiments. Give them a clear timeframe to run, then evaluate the results and decide whether to iterate or pivot. This approach forces discipline and ensures that you don’t waste resources chasing an idea that’s going nowhere.

Know When to Grit and When to Quit

Not every experiment will succeed, and that’s okay. What’s more important is knowing when to push through and when to walk away. This requires two key things: rigorous measurement and clarity of vision. Without clear metrics, it’s easy to get lost in the details and convince yourself that progress is happening when it’s not. And without a strong vision, you won’t know if your experiment is aligning with the broader goals of your customer or business.

“When you have a new idea that hasn’t been done before, you need that conviction, that blind faith that it could work.” - Andy Jassy

Imagine you are working with a customer who is migrating their on-premise infrastructure to AWS. The customer wants to experiment by moving a few legacy applications to the cloud to test for improved performance and cost savings. You propose an approach using AWS Lambda for serverless compute and Amazon RDS to replace their on-premise database. Your hypothesis is that these services will not only reduce operational overhead but also improve scalability and performance, while keeping costs manageable.

You start by migrating one of their smaller applications. You set up AWS CloudWatch to monitor performance and cost metrics, ensuring you have visibility into resource usage. You also use AWS Cost Explorer to track spending, and AWS X-Ray to trace the performance of your Lambda functions. The first iteration shows mixed results: while Lambda scales effectively under load, the performance of RDS isn’t matching the customer’s on-premise database, and costs are rising faster than anticipated due to higher-than-expected Lambda invocations.

At this point, you need to decide: should you grit and continue troubleshooting, or is it time to quit and pivot? The key here is your measurable success criteria. If the customer’s primary goal was to reduce costs, and you’re seeing little progress in that direction, it might be time to experiment with a different approach—perhaps using EC2 instances or Amazon Aurora instead of RDS to optimize both cost and performance. On the other hand, if the goal was scalability and serverless still holds promise, you could tweak your architecture and run more tests.

In this scenario, the experimental mindset comes into play: by breaking the migration into smaller, testable steps and continuously measuring the results, you can quickly determine whether to move forward or pivot to a new solution. You could also present your customer with multiple options, allowing them to choose what works best for them (don’t give them too many choices though - that may end up causing analysis paralysis!).

In today’s fast-paced environment, the ability to fail fast is crucial. Failing fast doesn’t mean giving up quickly - it means making data-driven decisions, iterating rapidly, and knowing when to abandon a path that isn’t yielding results. This is how an experimental mindset helps you optimize your time and resources, while continuously learning and refining your solutions.

Computers have been designed with this same mindset. Take an AI agent developing a computer vision system like the infamous “hot dog, not hot dog” app from Silicon Valley (which we were going to link here, but there’s lots of ahem… choice words). Early failures may occur because the initial convolutional neural network (CNN) struggles to differentiate subtle features between objects. To fail fast, the model might experiment with varying kernel sizes or deeper layers to capture more complex patterns. It may also apply data augmentation such as rotating images, adjusting brightness, or flipping the inputs to create a more robust dataset. By failing fast at each step, whether it’s the CNN architecture or the training strategy, the AI identifies what doesn’t work, refines its’ approach, and improves its predictive power. Switching back to you: it doesn’t matter whether you’re designing a solution architecture or building a GTM motion or even fine-tuning an AI system, the key is to treat each iteration as a learning opportunity. Like the AI tweaking its model, you need to be ready to adjust your strategy based on real-world feedback. Experimenting with different solutions lets you gather data, iterate quickly, and pivot before investing too heavily. Failing fast isn’t just for machines, it is a mindset that drives continuous improvement.

For those that are fans of the Agile Manifesto, you’ll note that it captures the essence of the experimental mindset with its opening phrase: “We are uncovering better ways of developing software by doing it and helping others do it.” This is experimentation in action - learning by doing, testing assumptions, and refining approaches based on real-world feedback. Agile ways of working are not about predicting the perfect solution upfront; they are about embracing uncertainty, running small, controlled experiments, and adapting quickly.

This approach aligns directly with the concept of one-way vs. two-way doors. In Agile, most decisions are treated like two-way doors—reversible and low-risk. Teams make iterative choices, learn from them, and adjust without the fear of irreversible consequences. The same mindset should be applied to your work: experiment, iterate, and pivot when needed. It’s not about getting every decision right, but about creating an environment where learning and improvement are continuous.

Conclusion

In our world of working with customers and partners, adopting an experimental mindset is not just a tactic; it is a necessity. It enables faster decision-making, more innovative problem-solving, and a culture of continuous learning. Run your experiments, track your metrics (but don’t worship them), and keep your curiosity alive. The more comfortable you get with uncertainty, the more you’ll thrive.

After all, experimentation isn’t about being right. It’s about finding out what works—and isn’t that the whole point?

Resources

Questions to jog your thinking:

1. What risks have I taken?
2. What would I do if I wasn’t scared?
3. What evidence is there that my skills are cutting edge?
4. (Add here if you know better ones)