During my time as a designer, I had the opportunity to collaborate with dedicated teams focused on enhancing our site through research and experimentation. During this experience, my proficiency in the language and methodologies of quantitative analysis, especially A/B testing, expanded significantly. This knowledge has greatly enhanced my capacity to collaborate with organizations committed to tangibly improving their services. In sharing my recent experiences, I aim to contribute to the ongoing discourse on how designers can effectively incorporate data and experimentation into their processes.
Traditionally, my design approach to information gathering revolved around qualitative methods such as heuristic and competitive analyses, interviews, and usability tests. LoveHolidays broadened my repertoire, making me more adept at integrating data as a crucial element of my design process. Credit is also due to the book “Designing with Data” by Rochelle King, Elizabeth Churchill, and Caitlin Tan, which laid the foundation for my understanding of the practical aspects and terminology related to this subject.
Initially sceptical of the data-driven design process outlined by our data and marketing departments, which involved numerous A/B tests on minor adjustments, I eventually grasped and expressed my concerns about the data-driven design. This led me to distinguish between data-driven, data-informed, and data-aware design.
- Data-driven: Focused on narrow paths, emphasising pure optimizations and efficiencies, often reflected in performance improvements and testing variations.
- Data-informed: Not confined to a narrow path, considers input beyond quantitative data, such as experience or instinct, and may involve A/B testing or structured usability tests.
- Data-aware: Understands the wide range and limitations of data collection, making decisions based on various methodologies, whether from stakeholder workshops, user interviews, or statistically significant A/B test results.
Why is this distinction crucial? Despite many businesses promoting themselves as data-driven, most designers rely on instinct, collaboration, and qualitative research methods. While I acknowledge a designer’s intuition as a significant asset, I argue for a data-aware approach to problem-solving. This approach enables designers to be valuable members of a data-informed team, fostering respect and collaboration within the organization.
- The Goal: Establish measurable goals, ensuring they align with ethical considerations and honest design practices.
- The Problem/Opportunity Area: Identifying areas for improvement based on data from various sources, such as user drop-off points, power user behaviour, support tickets, surveys, and user research.
- The Hypothesis: Crafting a testable solution with a well-structured hypothesis that includes user groups, changes, expected effects, rationale, and measurable results.
- The Experiment: Conduct experiments that control variables, represent the hypothesis, align with organizational requirements, and are meaningful and ethical.
The testing strategy involves an exploratory mindset to validate ideas and gain insights whether exploratory or evaluative, experiments contribute valuable information for future design decisions.
To illustrate the process, when working as a consultant we set goals to increase revenue, identified a problem/opportunity area related to the pricing structure, formulated hypotheses, conducted experiments, and analysed results.
In conclusion, while each project and team may have its unique approach, following certain steps—utilizing data triangulation, documenting structured hypotheses, communicating a testing strategy, maintaining big ideas, and starting with small tests—can benefit designers. Understanding the concepts and framework outlined can help designers communicate effectively, deliver customer-centric work, and align with modern organizational leadership. My exposure to this process has been invaluable, and I look forward to further developing my knowledge and sharing insights in the future.