If, like me, you’ve been to a fair few digital marketing events over the years you like to pick through the talks in your head and try to weave a coherent narrative thread out of them. Sometimes it’s harder than others. As a young whippersnapper, I remember attending an IAB Engage event (2006) in which advertising big-cheeses Michaelides and Bednash came out and declared ‘Content is King’ only for the next speaker to come out and announce that, no, actually ‘Context is King’ instead. Well, make your bleedin’ minds up. Sometimes events also have the occasional talk title like “SEO is Dead”, “CRO is Dead”, or “The Horse and Cart Is Dead”. Sometimes they’re right (Henry Ford).
So here’s my take on the main themes from Experimentation Elite from my stage-eye view:
Experimentation isn’t new. Historic examples included the British Navy’s 18th Century battle against scurvy from Stephen Pavlovich and a lucky escape from adding sulphuric acid in our Margaritas*, plus Craig Sullivan channelling James Burke’s “Connections” (younger viewers can Google it) in discovering the brilliant evolutionary line from carrots** to the iPhone.
Experimentation isn’t even new to marketing. Claude Hopkins was copy testing in the 19th Century. What is new is that digital interfaces enabled marketers to access more customer data at a faster rate in real time to make quicker and more accurate decisions. This is now infiltrating all parts of a company.
To evolve the company culture of experimentation the CRO practitioner has to be prepared to let go. Product teams and engineers should be running programs. CROs really should just be custodians of the governance and processes (the excellent Stewart Emhoff from RS).
It’s possible to move through the maturity gears quickly with the right expert guidance (Creative CX and there fast-growth programme at YNAP, Divya Isaiah’s reflections on growing experimentation teams and programmes)
In terms of maturity there are now three main types of practice. Experimentation 1.0 is centralised groups of practitioners or a single practitioner. At this stage, how do you make progress? Test velocity has been the historic milestone (Bhavik Patel). It’s still OK to have this as an objective and pick off the easier to reach learnings.
Experimentation 2.0 is a hybrid model, perhaps with practitioners in combination with consultants. We’re now finding it harder to find ‘winners’. Ton Wesseling showed how higher test velocity is correlated with weaker web experiments. So the focus should be on better validation of the solution (multiple great examples of this from Emma Travis at Speero and always-on research plus Eden Bidani with filtering through better copywriting). As experimenters we’re also now stepping out of our comfort zone of digital only experiments.
Experimentation 3.0 is the centre of excellence model. This is decentralised. Brands here are experimenting everywhere. CROs are not even practitioners any more- tests are done by default across all areas of the business. There aren’t that many organisations with a culture like this. Ellie Hughes, Tim Stewart, Jono Alderson and Bhav Patel chewed over the experimentation gap between the few gold standard companies and everybody else.
The next stage is looming large, trailed by what Jono Alderson called the forthcoming ‘Cookiegeddon’, with Chrome due to eliminate third party cookies in 2023 and GA4 designed to rely increasingly on modelled rather than actual user numbers. Experimentation 4.0 could be any combination of (a) only testing on your signed-in users and extrapolating results to all users- G-Szpalerka-Denison was excellent on how this is working within Asos app testing program (b) testing with a modelled sample of visitors rather than real actual user numbers (c) leaving AI and machine learning to make guesses on the fly and implement changes in real time (Sandeep Shah aluded to this in his examples of relevancy in navigation cues) (d) experimentation leaving the digital sphere altogether (unlikely).
Experimentation Elite showcased the brightest and the best sharing the best ways of working so that collectively we can take this thing forwards. So thanks to Stephen Pavlovich, Stewart Ehoff, Emma Travis, Chris Gibbins, Ibrahim Lawal, Paul Wainwright, Bhavik Patel, G. Szpalerska-Denison, Divya Isaiah, Eden Bidani, Sandeep Shah, Tim Stewart, Ellie Hughes, Jono Alderson, Ton Wesseling and Craig Sullivan.
*The Royal Navy ran experiments on scurvy prevention including cider, sulphuric acid and lime juice
**Friedrich Reinitzer discovered that certain derivatives of cholesterol (i.e. acetate, benzoate, etc.) extracted from carrots were in liquid crystal form paving the way for the LCD.