The anticipation of the sunsetting of Google’s Optimize 360 has come and gone so it’s time to consider next steps and how to evaluate alternative experimentation solutions for your business, particularly for experience optimization (EXO) efforts. There are a multitude of enterprise solutions with a variety of features but rather than tell you what options you have, we’re going to break down how to evaluate a vendor or what you may need to know in order to make an informed and meaningful decision for your business – beyond just price.
Assess Your Business Goals, Resources and Technical Requirements
While all A/B/n testing platforms will adequately perform basic split testing, different vendors’ tools have different capabilities when it comes to more advanced targeting and personalization. Therefore, it’s important to define what your specific goals are for your business in order to make an informed decision about technology tools for your organization. While you can certainly specify goals as increasing conversions, there are more advanced goals such as improving user engagement, determining how that engagement will be measured, determining whether that metric will be a multi-page metric or a per-page metric, etc. that can all contribute to the need for one platform over another.
It’s also necessary to consider what platform your website or application is built on. Are there are any specific limitations or requirements that the software needs to accommodate—again, as not all A/B/n testing platforms are created equal in this regard?
The amount of traffic that your website sees is also a critical factor as that will determine how robust of a platform your organization may require. A/B/n testing platform vendors will promise lower sample size requirements due to the fact that their (and most all) testing platforms these days use a Bayesian approach, but a Bayesian approach does not in general solve for low volume.
Lastly, your team implementing the testing platform plays a pivotal role and will determine the type of platform that will be ideal for your organization. The number of team members, skill level and technical proficiency all play a part. It is a key fact that different A/B/n testing platforms present different challenges with respect to onboarding and ease of use.
Determine What Features Are the Highest Priority
When evaluating experimentation platforms, developing a scorecard can help you objectively evaluate and compare different options. Assess the current maturity of your organization, what features you may be ready to utilize and what may be too sophisticated. You don’t want to waste money on features that your business or your team isn’t capable of or ready to implement. Additionally, consider features that will be useful over the next one to three years. The shorter the term, the higher the price, but also, don’t think too long-term as you may also outgrow the tool.
Here we present a basic scorecard, but this is only a starting point. This needs to be adapted to the organization in question, the organization’s specific goals, and the organization’s existing resources and willingness to bring on outside agency support.
In this sample scorecard, you can assign an importance level to each feature (e.g., High, Medium) based on your organization’s priorities. Then, assess each testing platform’s capability for each feature. You can use a scoring system (e.g., Excellent, Good, Fair, Poor) or a numerical scale to rate the platforms.
After filling out the scorecard for each platform, you can tally up the scores for each feature and come to a more informed decision about which platform aligns best with your needs. Remember that the importance level you assign to each feature should reflect your organization’s unique goals and requirements.
Consider the Level of Training and Support Needed to Fully Integrate A New Experimentation Solution
Different experimentation platforms offer varying levels of support to assist their clients in executing effective testing campaigns. Support may range from little to no support or basic technical assistance to more robust comprehensive strategic services. The level of support needed will depend on a variety of factors including previous testing experience, current internal resources, your operational model (COE, Decentralized, or Centralized) and end-to-end integrations. We’ll examine these a bit more and how each one may impact the level of vendor support you require. It’s also important to note whether the vendors charge extra for support or whether it’s included as part of the package.
Your Operational Model Can Indicate How Your Team Will Utilize Support
A Center of Excellence (CoE) operational model is a team or entity that provides leadership, best practices, research, support and training for a focus area or particular area of interest. Under this strategic approach, it’s likely that your testing team will have a better handle on testing tools, approach and best practices since their focus is on specialized expertise. So, they may not need as much training or focused support. They may see the vendor more as a partner or leverage their expertise for more advanced testing knowledge or techniques. They will utilize the support team as a way to stay on top of new trends and updated technology.
In a decentralized model decision-making is distributed across teams or departments. In this circumstance, each team may require customized support based on the specific needs of the team. Communication with the vendor’s support team would be at the department level alleviating the need to go through senior management or a centralized department and the team may look to the vendor’s support resources for specific or targeted assistance, guidance or troubleshooting.
Conversely, a centralized operational model means that one team makes decisions on behalf of the entire organization. Using this approach, decision making is more process driven or transactional so this type of company may rely on a support team strictly for technical support or assistance in how to execute standardized testing. Since resources are centralized, it’s likely they may not have a knowledgeable team dedicated to testing efforts so they may look to the vendor’s support team for more hands-on assistance with testing strategy and campaign execution.
The Complexity of End-to-End Integrations Can Impact the Level of Support Needed
End-to-end integration is the seamless connection and interaction between systems and processes within an organization – both upstream and downstream. Upstream integrations involve a connection between your experimentation platform and systems that provide data to that platform. For example, a customer data platform (CDP) or analytics tools like Google Analytics are both upstream integrations. If utilizing these types of tools, it may be important to have a support team to help with interpreting the data, set up proper tracking, troubleshooting data discrepancies or ensuring data consistency. If utilizing a CMS or CRM system, it may be important to have support in order to ensure smooth integration between the testing tool and the systems in order to personalize user experiences in the user tests.
Downstream integrations refer to connections between your testing platform and systems impacted by the result of your tests such as conversion tracking, personalization or marketing automation. Support may be needed to ensure that test results are reflected in downstream systems and actions are taken based on those results. In both circumstances, when selecting a vendor, it’s important to weigh your integration needs and understand how the vendor’s support services align with your integration requirements.
There Are Direct and Indirect Costs Associated with Implementing An Experimentation Program
Many companies simply evaluate vendors based on cost. They may get three estimates and go with the lowest cost proposal. But, as we’ve explained thus far, that really isn’t optimal for selecting a partner as there are so many other factors that contribute to determining which A/B/n testing tool is best for your particular needs. Even when it comes down to cost, there are both direct and indirect costs that need to be considered. Here are some true costs to keep in mind:
Software costs
Experimentation vendors may charge a subscription fee based on factors such as number of campaigns, number of users or website traffic so it’s important to understand the pricing structure.
Implementation costs
Running an experimentation program may require integration with existing MarTech systems such as CRM, CMS, etc. This could involve additional resources which in turn translates to additional costs.
Test-Development Costs
Your team will be responsible for brainstorming, creating and planning meaningful test variations, then setting up the experiments, monitoring and analyzing the results, which takes time and resources. There is a cost associated with these resources since this time can’t be spent doing other things for the business.
Scaling Costs
If your business grows, your testing needs could increase which means you may need to upgrade your plan or add more advanced features.
Opportunity Costs
If ramping up slowly in your testing approach (perhaps you have limited resources, want to minimize disruption or want to take the time to learn more about testing before diving in) you may delay results or conversions which means missed revenue, fewer optimizations and slower growth. This may save you money upfront, but you’ll be sacrificing potential gains in the short term while you learn.
Outside-Consultant Costs
Conversely to opportunity costs, if you decide to speed up the process by hiring an expert to help with testing efforts, you will incur higher upfront costs, but it will yield faster and more impactful results. Their expertise will help you avoid common mistakes, design more effective tests and require less time to see growth and conversions.
When evaluating vendors, it’s imperative to consider your breakeven point and the hidden intangibles of experimentation. Meaning, evaluate the point where the financial benefits of a platform outweigh the costs associated with that platform. Recognize both the direct and indirect costs as well as the monetary and non-monetary benefits and drawbacks that come with experimentation.
Determine the Ideal Return on Investment Over the Lifetime of the Agreement
As you process the information on how to evaluate experimentation vendors to find the best fit for your organization, it’s critical to define the ideal return on investment (ROI) that you aim to achieve over the lifetime of your engagement. The journey of selecting the right vendor goes beyond just technical features and costs; it involves aligning your testing objectives with tangible business outcomes.
Calculating ROI encompasses assessing both costs and benefits associated with your testing campaigns. Below are seven steps to effectively determine ROI of your experimentation platform:
- Align With Business Goals – Understand how your testing campaigns directly impact achieving your business goals (i.e., boosting sales, increasing conversions, improving user experience).
- Account For Actual Cost – While software cost isn’t the only determining factor when making a decision, it does play a part once you’ve determined whether the features, support and other factors meet your requirements.
- Measure Financial Benefits – What monetary benefits will the business see as a result of the testing?
- Account For Resource Costs – Factor in the time required for your team to design, set up, execute and analyze data from the tests as well as ongoing monitoring and optimization.
- Weigh Opportunity Costs – Recognize the opportunity costs stemming from delayed decisions, slower optimizations or missed revenue due to suboptimal testing strategies. Weigh these costs against potential gains.
- Consider Indirect Benefits – Benefits such as enhanced user experience, increased customer loyalty, improved brand perception or optimized marketing efforts are intangible gains that can impact ROI.
- Learn And Improve – As campaigns progress, iterate and optimize. Also, as time goes on, teams will also learn, become more familiar with the platform and work more efficiently which will save time and in turn, money, boosting ROI.
Following the Sunsetting of Google Optimize 360, Understanding the Distinctions Experimentation Vendors is Essential
Experimentation empowers businesses to make strategic, evidence-based decisions and optimize user experience in a data-driven environment. Engaging a vendor that aligns with your business goals, offers the right mix of features and support for experience optimization (EXO) and demonstrates a solid potential for ROI will likely contribute most effectively to your business’s overall success.