Flicker-free A/B tests.
Why a tested page can jump under the visitor’s eyes, and how to stop it. Work the lantern to see both.
Turn your phone
Move the slide · Open the shutter
A visitor has just asked for the page.
Same two moves. Only the order changes.
It can change the result of the test.
A page that jumps gets noticed, moves things under the finger and feels broken. A visitor who saw that does not behave like the others, so the test measures the flicker as well as your change.A page that jumps gets noticed, moves things under the finger and feels broken. A visitor who saw that does not behave like the others, so the test measures the flicker as well as your change.
What it means for you.
I set your clients’ tests up so that the change is made before the page is shown. Their customers never see a page jump, and your tests measure your change and nothing else. I check it on each store before the test goes live.I set your tests up so that the change is made before the page is shown. Your customers never see a page jump, and your tests measure your change and nothing else. I check it on your store before the test goes live.
The details, if you want them.
Pull a drawer.
And on a slow connection?
I set a limit for each test. If the change is not ready in time, the visitor gets the original page: nobody is left in front of an empty screen.I set a limit for each test. If the change is not ready in time, the visitor gets the original page: nobody is left in front of an empty screen.
And when the test is over?
When a version wins, I can build it into the theme and remove the test. From then on nothing is changed on the fly: there is no flicker to avoid, and no wait.When a version wins, I can build it into the theme and remove the test. From then on nothing is changed on the fly: there is no flicker to avoid, and no wait.
And a visitor who comes back?
They see the version they saw the first time, and are counted once. The same person never counts twice in the result.They see the version they saw the first time, and are counted once. The same person never counts twice in the result.
And if something goes wrong?
You can switch a test off yourself, at once. Every visitor then gets the original page.You can switch a test off yourself, at once. Every visitor then gets the original page.
And what does it cost?
Without flicker, the page is shown a little later. How much depends on the test, on the size of the change and on the visitor’s connection. I measure it on each store before the test goes live.Without flicker, the page is shown a little later. How much depends on the test, on the size of the change and on the visitor’s connection. I measure it on your store before the test goes live.