Live-data prototypes are a little tougher to explain, but they are absolutely critical and the cost of producing them is dropping rapidly so I’m loving them more all the time. The main purpose of a live-data prototype is to actually prove something – normally it’s to prove whether an idea (a feature, a design approach, a workflow) really works. In order to know this, we typically need to do two things. First, we need the prototype to access our real data sources – like actually search our live inventory and show products that are really available right now. Second, we need to be able to send live traffic, in quantity, to the prototype.

The key is that we sure don’t want to have to build, test and deploy a real product in order to do this. That would take far too long, and cost far too much, and yield huge waste. And we don’t. A live-data prototype is a very limited implementation – typically just the critical use cases, and none of the “productization” that’s normally required like full use cases, test automation, SEO work, internationalization and localization, performance and scalability, etc. A live-data prototype is a fraction of the effort, but you get big value. You do have to keep in mind two big limitations. First, this is code so it requires your developers to create the live-data prototype and not your designers. Second, this is not a product, and you can’t run a business with it, so if the tests go well, you’ll still need to allow your engineers to take the time to productize the code.

[...] Normally we’ll test a live-data prototype in an A/B test, but we can also do an opt-in test or an invite-only test. The key is that real users will use the live-data prototype for real, and this will generate real data (analytics) that we can compare to our current product to see if this new approach actually performs better.