Archive for the ‘Experiments’ Category

Experiment-driven, data-driven approach to web design

August 30, 2007

The book Hard Facts talks about the importance of making decisions based on evidence and data (“experiment-driven, data-driven mind-set”).  At this point in the book the authors are talking about how fully Yahoo! embraces this approach:

“Yahoo! Inc is skilled at running experiments and learning from them, as well as building a culture that emphasizes evidence-based management. Usama Fayyad, chief data officer at Yahoo!, points out that because its home page gets literally millions of hits an hour, the company can design rigorous experiments that yield results in an hour or less — randomly assigning, say, one or two hundred thousand visitors to the experimental group and several million to the control group … Yahoo! conducts experiments and uses the results to enhance company revenues and profits.  Much of this can be done very quickly; sometimes, results can be seen within minutes of tweaking something on the homepage or in Yahoo! Mail. This means there is no reason to spend time discussing which variation to explore or what design opportunities to pursue — it is often cheaper, easier, and faster to simply try all of them and learn what actually works. Yahoo! typically runs 20 or so experiments at any time, manipulating things like color, placement of advertisements, and location of text and buttons. These little experiments can have big effects, like the one run by Nitin Sharma, which showed that simply moving the search box from the side to the center of the home page would produce enough additional ‘click-throughs’ to bring in about $20 million more in advertising revenue a year … Yahoo! has the mind-set that says, Instead of debating which screen design looks best, or which placement of content and which choice of specific content works best, we’re going to try it all and see what works.”

Hard Facts by Jeffrey Pfeffer and Robert I. Sutton