The Process

We conducted a variety of tests and analyses to allow us to generate insights for this client that would ultimately lead to a more effective online presence. This case study, however, focuses in particular on the preference testing we conducted at two milestones, namely before our design mockups and after our design mockups; and the actual results post-launch.

Pre-Mockup Testing

We conducted preference testing between Chas Roberts and three of its key competitors. The testing was conducted online using a panel of respondents that met our target demo criteria. The preference testing yielded the following results, with the Chas Roberts site being the decisive loser.

Note that the position of each competitor home page was rotated for each successive user taking the preference test to avoid any systematic bias to placement.

Statistical testing indicated that probability that Chas Roberts lost to Mr. Rooter by mere chance was <1%  (p<.001).

Post-Mockup Testing

After completing our full research and user experience analysis and strategy, we recommended wireframes to the client which were approved and then led to us developing mockups. However, before proceeding to development, we conducted a second round of preference testing with a new group of respondents that met our target demo requirements. The results this time were dramatically different.

Note that the position of each competitor home page was rotated for each successive user taking the preference test to avoid any systematic bias to placement.