Mark noticed this Businessweek article on how Yahoo! mined their user-click data to inform the redesign of their home page (How Yahoo! Gave Itself A Face-Lift):
To avoid design by committee, Yahoo deferred almost every decision to an impartial judge: data generated by users’ clicks. “We have this culture of data,” Bhat explains. “It is the biggest enforcer of honesty.” If sales wanted an ad smack dab in the front page’s prime real estate, the company would whip up a page to those specifications, serve it to actual users, and record their clicks. If traffic increased, great. If not, it was back to the drawing board. The refreshed home page went live in September. Now, with a multimillion-dollar ad campaign to promote the redesign under way, the company is keen to see whether it has truly created a page based on what users like rather than what Yahoo wants.
…
Yahoo made a commitment to harnessing its trove of user clicks in 2004 when it acquired DMX Group, a data mining consultancy founded by former Microsoft Corp. (MSFT ) researchers. Now called Strategic Data Solutions (SDS), the department has a daunting task: combing through the 10 terabytes of data that Yahoo users generate daily by clicking links (the equivalent of all the text in the Library of Congress), plucking out the relevant bits, compressing it, and storing it. So far, Yahoo has enough user data to fill more than 1,000 Libraries of Congress.
Of course, all that information would be useless without a way to make sense of them. Before Yahoo bought DMX Group, a simple test of how users interact with a page required help from technologists and a month of preparation. Now nearly all employees have access to easy-to-use software tools that can run tests over a few hours or days. Along with providing the tools, the SDS department has worked to spread the gospel of data. “We say: Use data to make decisions. Don’t make decisions based on a fad or what your competitors are doing,’” says Bassel Ojjeh, vice-president for SDS.
To be clear, Yahoo! didn’t rely entirely on data alone to make all of their crucial design decisions:
What Yahoo learned often belied initial impressions. Throughout the redesign, the company used a blend of focus groups, one-on-one interviews, test pages, and data mining. “What people say they want isn’t always what they actually click on,” Bhat says. In focus groups, users consistently said they wanted serious world news. “I don’t want Britney Spears anywhere on my page,” Bhat recalls one user saying. “What if my boss came by and saw?” But when Bhat’s team studied users’ clicks, world news got little attention, while Britney Spears stories ranked among the most heavily trafficked.
The mixed messages led to important insights. In the end, Yahoo kept world news prominent on the front page because users feel secure knowing that it’s easily accessible, even if they don’t often click it. Conspicuous placement also went to entertainment, which draws heavy traffic from people seeking a diversion at work. By contrast, seemingly work-related content such as finance gets ample use in the evening when people pay bills and manage personal portfolios.
Another gem unearthed through data mining: Small changes can make a big difference. The redesign team was excited about a new feature called Personal Assistant, which lets users hover their pointers over icons to see preview boxes of content such as e-mail. “We knew this was going to be the wow’ element of the page,” Bhat says. But the data showed that users were less than wowed. Turns out the preview boxes opened too quickly, an unusual peeve in this caffeinated, wired world. So the team began fiddling with the speed at which the preview boxes appeared and introduced a slight delay. Bingo.
Although Yahoo’s front-page redesign is finished, the testing is not. “There’s always some test running,” Bhat says. “It’s part of our DNA.” Now if only Yahoo could collect as much data about its advertisers’ spending habits.