Eric Reis wrote a great book, The Lean Startup, that I recommend everyone read. Before I proceed to talk shit about some of the methodology, I want everyone to know I utilize the lessons of this book everyday to improve the businesses I work with. I am just explaining two points of caution.
One of the main themes in the book is using the scientific method (controlled testing and data analysis) to improve your business. The idea is that every action a company takes should be measurable by non-vanity metrics, i.e. metrics that drive your business model like customer aquisition cost, churn rate, or number of users using your service daily. The action can then take place and be tested. If the action or strategy performs well in non-vanity metric terms, the company integrates the strategy. The firm then moves on to an additional test. Its about fine tuning the business engine. As an Ex-Quantitative Hedge Fund Trader, I love the idea of letting the data drive the business strategy. However, that experience also taught me two important issues. They apply well to the Lean Startup methodology. The first is the data mining issue. The second is solving for a local maximum when you think you have solved for a global maximum.
Reis does not emphasis that every business experiment should have a rational theory behind it. This is a mistake. Take the example of A/B testing of single words on a content website. If you perform thousands of single word tests, you will eventually find amazing results. Results like if you use the word “midnight” instead of “12:00AM” that your conversion rate will go up by 1%. This is a major issue in statistics called data mining. When you start to perform lots of experiments not based on rational theories, you will find statistically significant yet meaningless results for a small set of experiments. If you perform those experiments again, you probably won’t get significant results on round two. Well thought out experiments that are based on rational theories return results that are more durable. The other reality is dumb experiments waste a huge amount of time. Seems obvious, but you see people testing things that make no sense by even large stretches of the imagination.
The local maximum issue is more problematic. I’ll explain by example.
The basic test here is how we are going to spend our advertising budget, Paid Search vs. SEO. The non-vanity metric is customer acquisition (CA) because we are an eCommerce site. CA and retention rate drive everything. We are currently focusing almost all our efforts and cash on paid search at Point A. We decide to perform a test of increasing our focus on SEO. We move to Point B during the test and measure a significant decrease is CA. Awesome! So our next test we decide to increase our focus even more on SEO. We move from Point B to Point C during the testing period. We measure a statistically significant uptick in CA. We scramble back to Point B. When it comes to this strategy we assume we are on an optimal point at Point B. However, we have just found a local maximum. A matter of fact, any incremental testing will drive us back to Point B. It is clear that this is sub-optimal when we look at the reality (which we don’t know as testers) shown in the graph. We want to be at the global maximum of Point D! (Note: I use global/local maximum when technically I’m talking about minimums. I do this because this issue as usually described as the “Local Maximum Problem”.)
It is important to realize that many elements of business have a graph like the one above. Local maximums are all over the place. Using SEO probably won’t start to payoff with low customer acquisition costs until you jump to putting most your focus on it. Vertical integration in manufacturing is the same. When you first start to integrate, profitability will go down. As you make massive integration moves profitability may drastically rise.
What is the solution? Use The Lean Startup methodology to constantly improve your business but make sure to take some long shots when tests don’t work. If focusing on SEO starts to hurt your business, try to test a larger change in focus towards SEO for a brief period. A way to solve this issue is described in Exploiting Chaos, a great book by Jeremy Gutsche, with a suggestion that every company should have a “gambling fund”. With a gambling fund, large and sometimes costly projects can be pushed out with no remorse and no intermittent testing. This is an effective way to find a global maximum out there that you are missing! The book is a great read for a bunch of other reasons too, and Jeremy is a great guy with some special insight over at TrendHunter.com. Not to mention he knows the best steak houses in Toronto.
The gambling fund topic is covered towards the end of this keynote: