Recently I led a panel discussion with twelve industry experts in Conversion Rate Optimization (CRO) and experimentation. The conversation addressed the nuances, challenges, and advancements shaping CRO as we progress in 2024. The panelists brought a wealth of experience and diverse perspectives, contributing to a rich exchange of ideas and future predictions. Here are the key takeaways from the discussion, offering a glimpse into the collective wisdom of leading CRO professionals.
The Unpredictable Evolution of Experimentation Mindsets Alters the Landscape of Business Strategy
As we kicked things off, panelists highlighted insightful shifts in organizational attitudes towards CRO and experimentation. The consensus was that there’s a growing understanding among executives and stakeholders about the value of these practices. Where detailed explanations were once necessary to convey the value of experimentation, there now seems to be widespread acceptance and knowledge of its benefits. Brian Massey, managing partner at Conversion Sciences, acknowledged this evolution, noting that by 2023, executives no longer needed the extensive justifications that were essential in 2021. While this change indicates a broader organizational shift towards embracing experimentation, he also highlighted that, although overall acceptance of CRO has increased, there is still a lack of understanding regarding the necessary skill sets for effective implementation.
Haley Carpenter, founder of Chirpy, echoed the same sentiments and concluded that although there’s a growing recognition of the value of experimentation, many companies are still struggling with its fundamentals. There is a noticeable gap between perception and reality. This disparity is more pronounced in some organizations than others. However, it underscores the importance of continuous education for stakeholders and the necessity of advocating for alignment at the C-suite level.
Gabriela Florea, CRO manager at Verifone, offered her perspective on the evolving approach of companies towards experimentation and their historical rigidity around statistics. She believes companies are increasingly focused on the velocity of testing and the learning aspect of experiments, even if they don’t always lead to immediate wins. Additionally, panelists agreed that clients are becoming more receptive to ‘losing tests,’ understanding that not every test iteration will lead to conversion or revenue growth.
Massey added that it is a pragmatic approach being adopted by some companies. For example, he shared that he had a client who valued the ability to test changes as a form of ‘design insurance,’ ensuring that new implementations do not adversely affect their operations or bottom line. He expressed that this approach is less about constantly finding big wins and more about risk mitigation and loss aversion. We’re seen the same thing with clients at Zion & Zion. For example, one of our clients in the travel and hospitality sector is keen to conduct an experiment aimed at assessing the impact of a specific technological investment on the overall user experience as a form of risk mitigation, not revenue growth. Additionally, we are working with a few clients who are undergoing complete website redesigns and experimentation serves as a vital tool to reduce the risks associated with such extensive redesign efforts.
Iqbal Ali, experimentation lead at Mild Frenzy Limited, Tracy Laranjo, an independent CRO consultant and Andra Baragan, founder of Ontrack Digital all confirmed a shift in their consultative work from purely conversion-centric research and A/B testing to a new focus that encompasses broader research, analytical insights, comprehensive site audits, and a focus on client retention. As consultants, this allows them to be more integrated into the organizations they work with.
These shifts are very positive for our industry, indicating a change in how businesses perceive the value of CRO and experimentation – focusing more on consistent, iterative learning and risk mitigation, and less on significant revenue growth.
The Transition from Tactical Experimentation to Strategic Foresight Marks an Unforeseen Journey
While the general maturity and shifts in the industry are positive, we’re still navigating a complex phase, and there are signs that it’s slowly carving out a strategic place at the decision-making table. This evolution, however, faces a critical challenge: the prevailing market emphasis on ‘growth’ often leads businesses to design for executives rather than end users. To sustain and advance this shift, it’s imperative that practitioners align with the C-suite. As Ben Labay, CEO at Speero, acknowledged, CRO typically exerts its influence on business decisions from the bottom up, leveraging user insights and data. Yet, the reality is that most business decisions are still dictated top-down. He advocates for a ‘metrics accountability system’ to ensure stakeholder alignment. While this approach is valuable, it’s only a part of the solution. True transformation requires a deeper alignment, extending to the executive level, not just among stakeholders.
If such alignment is achieved, we can expect a gradual dissolution of traditional CRO roles within organizations. The function’s value will become increasingly integrated and less distinguishable from other business functions. Jonny Longden, director of digital experiences at Journey Further, gave a relevant example with Amazon. He noted that Amazon does not have a designated ‘head of email’ due to email management being an integral part of their business operations. They recognize it as a necessary function without needing specialized oversight. Similarly, for CRO to truly thrive within an organization, the boundaries between it and other functions should be less distinct. The more seamlessly CRO is woven into the fabric of an organization, the greater its efficacy and contribution to the overall business strategy. This integrated approach is not just about blurring lines but about fostering a holistic, collaborative environment where CRO and experimentation is an intrinsic part of the business landscape, driving growth and innovation from within.
Rising Marketing Costs Push Businesses Towards PLG Strategies
With regards to adoption and growth, Collin Crowell, VP North America at Kameleoon, confirmed a significant transformation in the business landscape. The increasing costs of marketing-led growth combined with the rising importance of product managers and engineers are steering corporate strategies in new directions. This change is propelling the Product-Led Growth (PLG) movement, which advocates for a more technical approach to business growth, centering on backend operations and enhanced collaboration between engineering and data analysis teams. With traditional marketing methods becoming increasingly expensive, more businesses are gravitating towards PLG, utilizing the technical expertise and innovative capacity of product teams to boost customer acquisition and retention.
Chris Neumann, founder and CEO of Cro Metrics, noted that tech behemoths such as Alphabet, Amazon, Apple, Meta Platforms, Microsoft, NVIDIA, and Tesla typically prioritize running experiments before deploying code. While such practices have long been commonplace among these firms, known as the ‘Magnificent Seven,’ their approach is a testament to the strategic value attributed to product development and engineering in fostering business success. As PLG gains traction, there’s a noticeable shift towards feature experimentation and server-side testing, effectively integrating CRO and experimentation into the very heart of product strategy. This shift signifies the transformation of CRO from a mere tactical tool into a pivotal strategic asset, playing a central role in driving organizational growth and innovation.
Chart the Unknown in Experimentation’s Future-Proofing Strategy
Paul Bernier, VP of product management at SiteSpect, a leading experimentation vendor, discussed a growing shift to server-side experimentation, which he views as a sign of industry maturity. He acknowledged that more product teams are adopting server-side testing to tackle more complex experiments than what client-side testing can offer. Crowell pointed out the continuing relevance of client-side testing, particularly for its ability to facilitate faster execution than server-side testing. They both see this trend as indicative of a maturing experimentation space. The integration of server-side testing into organizational processes signifies a deeper, more ingrained approach to CRO and experimentation, ultimately delivering more value in the long-term.
Bernier further discussed how his team is closely monitoring the impact of Intelligent Tracking Prevention (ITP) on their clients’ web traffic. This focus is particularly on users of browsers like Safari, which are significantly affected by ITP’s cookie limitations. In cases where clients are notably impacted by ITP, especially those in regions with stringent privacy regulations like Europe, a shift to server-side solutions is being considered. Such a move would ensure more consistent and reliable data collection in scenarios where client-side tracking faces challenges. Additionally, in heavily regulated industries, such as healthcare and finance, these considerations greatly influence the way experimentation platforms are designed and handle test data.
Not all server-side migrations are ITP or privacy related. Maya Goradia, experimentation lead at ZD Analytics confirmed that several of her technology clients are moving their Google Tag Manager (GTM) from client-side to server-side in an effort to secure significant improvement in data quality and the effectiveness of experimentation.
Navigate AI’s Uncharted Waters in Modern Experimentation
The use of Artificial Intelligence (AI) in 2024 was also discussed, revealing a mix of current applications and a pragmatic future prediction. Many are using AI in expected ways, like generating ideas and summarizing meeting notes which others are using to generate new ideas in unfamiliar or niche industries. Then user testing is utilized to validate these ideas before suggesting A/B tests to clients. This approach of casting a wide net with AI-generated ideas, followed by refinement through cost-effective research, and then validation before making significant business decisions, exemplifies a strategic use of AI in the current business landscape.
Crowell believes that there are challenges due to the increase in experimentation ideas generated by AI. He noticed a trend of mid to large size companies looking for solutions to catalog insights. This strategy, similar to a composable data warehouse, aims to document experimentation hypotheses, test results, and learnings for efficient AI querying. While this use case might not be too surprising, its commonality in current contexts indicates how AI will likely enhance experimentation and business knowledge moving forward. While this direction is not necessarily where some platforms are headed, it’s a topic increasingly discussed with clients and prospects. This trend suggests a future in which AI not only generates ideas but also aids in organizing and deriving value for the future.
Many agreed that AI’s broader impact indicates an amplification of existing challenges in valuing, prioritizing, and implementing ideas. As AI escalates the volume and complexity of ideas across various business units, the need for robust communication skills, business acumen, and customer understanding becomes increasingly important for all involved in experimentation. This progression also hints at a future where non-technical team members might be more involved in AI-driven experimentation processes. Thus, AI is not merely a tool for enhancement but also a catalyst for a more inclusive and collaborative approach to business strategy and experimentation, potentially driving the transformative change the industry seeks.
Embrace the Transformation in CRO and Experimentation in 2024
As we conclude our exploration of 2024’s experimentation trends and predictions, it’s evident that the landscape of Conversion Rate Optimization (CRO) and experimentation is undergoing a dynamic transformation. This evolution, driven by a deeper integration of experimentation into business strategies and an increased emphasis on risk mitigation and iterative learning, signals a pivotal shift in the industry. The insights and experiences shared by our panel of experts not only illuminate the path forward but also underscore the importance of adapting to these changes. As organizations continue to navigate this evolving terrain, the role of experimentation in shaping business success becomes ever more crucial, heralding an era of innovation and strategic growth.
If you’d like to contribute what you think the top 2024 Experimentation trend will be, join the conversation live on LinkedIn.