Well-documented cases of A/B testing include campaign fundraising, Internet marketing, and the Lean Startup methodology.
User research is so inexpensive and easy relative to the effort and cost of building a useless feature in code that, in almost every case, we shouldn't prioritize a feature without some form of validation.
In other words, two-thirds of features either have a negligible impact or actually make things worse.
If we are not performing user research, the odds are that two-thirds of the features we are building deliver zero or negative value to our organization, even as they make our codebase ever more complex, thus increasing our maintenance costs over time and making our software more difficult to change. Furthermore, the effort to build these features is often made at the expense of delivering features that would deliver value.
integrate A/B testing into the way we design, implement, test, and deploy our features. Performing meaningful user research and experiments ensures that our efforts help achieve our customer and organizational goals, and help us win in the marketplace.
Fast and iterative A/B testing is made possible by being able to quickly and easily do production deployments on demand, using feature toggles and potentially delivering multiple versions of our code simultaneously to customer segments. Doing this requires useful production telemetry at all levels of the application stack.
How we can frame hypotheses in feature development in the following form:
We believe X will result in Y and we will have confidence to proceed when Z
Adopting an experimental approach to product development requires us not only break down work into small units (stories or requirements) but also validate whether each unit of work is delivering the expected outcomes. If does not, we modify our road map of work with alternative paths that will actually achieve those outcomes.