The Client

Chas Roberts is the lead home service provider of HVAC equipment and plumbing services in the Phoenix Metropolitan Area.

Zion & Zion has served as the client’s Digital Agency of Record for the past several years.

The Challenge

The Phoenix Metro Area is one of the most competitive in the country when it comes to HVAC and plumbing home services companies. The market is highly saturated with aggressive competitor spending on all forms of media. To maintain and extend Chas Robert’s market leadership, we developed a new website for the company to complement our other digital strategies.

The Process and Results

We conducted a variety of tests and analyses to allow us to generate insights for this client that would ultimately lead to a more effective online presence. This case study, however, focuses in particular on the preference testing we conducted at two milestones, namely before our design mockups and after our design mockups; and the actual results post-launch.

Pre-Mockup Testing: We conducted preference testing between Chas Roberts and three of its key competitors. The testing was conducted online using a panel of respondents that met our target demo criteria. The preference testing yielded the following results, with the Chas Roberts site being the decisive loser.

pre mockup testing

Results from the preference test we ran in the pre-mockup stage.

Note that the position of each competitor home page was rotated for each successive user taking the preference test to avoid any systematic bias to placement.

Statistical testing indicated that probability that Chas Roberts lost to Mr. Rooter by mere chance was <1%  (p<.001).

Post-Mockup Testing: After completing our full research and user experience analysis and strategy, we recommended wireframes to the client which were approved and then led to us developing mockups. However, before proceeding to development, we conducted a second round of preference testing with a new group of respondents that met our target demo requirements. The results this time were dramatically different.

post mockup testing

Results from the preference test we ran in the post-mockup stage.

Note that the position of each competitor home page was rotated for each successive user taking the preference test to avoid any systematic bias to placement.

Statistical testing indicated that probability that Chas Roberts beat each of the competitors, merely by chance was as follows:

Mr. Rooter                    ≈10%           p=.1096

George Brazil               <1%             p<.0001

Benjamin Franklin      <1%              p<.0001

The ultimate measure of any marketing action however, including our development of a new site for Chas Roberts, is the impact that the investment has on the client KPIs—in this case, lead generation. To that end, we measured the impact on leads generated by the website through use of call tracking numbers. We found that in the weeks leading up to the website launch, year-over-year leads generated were down 5.5%, while in the weeks that followed the website launch, year-over-year leads were up 25.0%.

In short, not only did our pre-testing reveal that Chas Roberts was at a competitive disadvantage with respect to its website, but our post mockup testing revealed that we likely were on the right track, and our final, post launch testing confirmed the superiority of the new site design.