@anonymous-piwik-user opened this Issue on December 4th 2008

Which of our landing pages are the most successful and convert our visitors better?

How often did we wonder if we have the right words and images on our landing page (the page where visitors arrive from your advertising)?[What if changing just one image on the page could increase the visitor conversion rate several times?[BR] We will never know this without testing.

The good news is that there is a technology that can help us quickly determine the best version of our landing page. It is called Split-Testing and it should be implemented in Piwik.

Piwik should randomly show different versions of the same landing page to our visitors. It should make sure that each version of the page is displayed the same number of times overall and it should even display the same version of the page to the same visitor (to make their experience with our site consistent).

With split-tests we should be able to quickly see which version of the page has a good conversion rate and which one is just a waste of the advertising money. Get rid of the unsuccessful version, keep the good one and our overall conversion will increase.

@robocoder commented on September 3rd 2010 Contributor
@robocoder commented on April 25th 2011 Contributor
@robocoder commented on June 2nd 2012 Contributor

WebSite Optimizer is being folded into GA.

@mattab commented on June 3rd 2012 Member

I think it was reasonnable for us to recommend using Website Optimizer, but now that they are forcing users to use GA for split testing, we should consider implementing similar functionality in Piwik :)

@mattab commented on September 21st 2013 Member

It would be fantastic to have A/B testing feature built in Piwik, as a plugin. If you are interested and could help us make it happen, please email at hello@piwik.org!

@nkuehn commented on March 6th 2014

my two cents in case you consider implementing A/B testing support (which would be awesome):

An extremely powerful way to realize the A/B testing capabilities but also much more would be to add a generic ability to compare segments. The new segment editor is awesome and could be leveraged even more.

A/B variant tags would then just be custom variables (typically visit-scope) like any other.

A feature like Google Analytics' capability to actively control which users get which variant from inside the analytics system is just for convenience, but not a core requirement IMHO. One can do the actual traffic split on the website server (or loadbalancer) or in the webisteJavascript as well.

@mattab commented on April 4th 2014 Member

Facebook has released their A/B testing tool: http://facebook.github.io/planout/ https://github.com/facebook/planout - it's written in python. Maybe we could get some inspiration from them.

@pattonwebz commented on April 25th 2014

I'm in agreement with nkuehn about it being unnecessary to have the ability to control variants from inside the analytics system.

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

I've been tracking split-test campaigns in GA for a while now using Events. Earlier today Piwik also introduced Event tracking (http://piwik.org/docs/event-tracking/) which I was then able to integrate with my exiting split testing framework (based on this javascript library on github: https://github.com/jamesyu/cohorts/).

Using almost exactly the same code as I use for GA tracking I am now able to track split-test campaigns with Piwik through with the Javascript API and Events. I'm using the following code which reports the same data to both GA and Piwik. First line is passing data to GA, second to Piwik.

_gaq.push(['_trackEvent', category, action, opt_label, opt_value, int_hit]);
_paq.push(['trackEvent', category , action, opt_label, opt_value]);

That code is called onInitialization of the test (consistency is handled by cookies before test initialization) to track how many times each test runs and then it's allso called a second time onClick passing a different label so that you can segment views from clicks.

@mattab commented on April 25th 2014 Member

Kissmetrics A/B testing product is interesting also: http://blog.kissmetrics.com/kissmetrics-ab-test-report/

@mattab commented on April 27th 2014 Member

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

It is not the "best place" if you are already using a A/B testing tool, but many people are only using Piwik on their website as "third party javascript" and would enjoy basic A/B tests feature in Piwik.

Maybe we could make it possible to use Piwik to measure A/B testing results from another tool? Users could use their favorite A/B testing tool to define and manage tests and variants, and send the data directly to Piwik (or Piwik fetch it from their API).

Our vision is to empower users with the toolset to measure and make changes on their website. Running A/B tests (create variants, serve them, measure performance, report results) is a key part of this vision :+1:

@pattonwebz commented on April 27th 2014

Hey Matt,

At least we agree that the 'best place' isn't directly within the Piwik platform, but it obviously wouldn't be a breaking change for me if it did include some functions for implementing tests.

I have seen the roadmap showing that A/B testing is to be implemented into the core - which is how I found this ticket - and I don't necessarily think it's a bad idea.

My primary reason for stating objections is not to prevent it being added (because it would be a greatly welcomed featured for a lot of users) but rather to note that some users would rather make use of a platform independent testing model that does not tie you directly to any one anayltics platform or the other. As such it would mean that the testing would need to be turned on/off at the provider end to prevent bloat in the tracking code for users who don't want it.

I get that having to use an external provider is not ideal but self-hosted libraries are also an option to negate the requirement of an external provider.

I guess what I'm really saying is that platform independence is important FOR ME but I get that it's not as important for others.

As I stated I already run tests and track them with Piwik using Events and I'd actually love to be able to help shape the integration of A/B testing to the core - which is why I commented on this ticket in the hope that someone from the core team would see it and I'd be able to make suggestions and push for my choice of perfect match features in they system ;)

@mattab commented on April 28th 2014 Member

Good we're having this discussion, it's useful!
Here are some thoughts on how it a new A/B Plugin could be broken down:

  1. let user define A/B tests and their variants via new UI and API. Variations include:
    • URL variations of same page new (mvp?),
    • let user create tests variants by inputting a HTML text (mvp?),
    • advanced UI to create test variants directly within the website like commercial A/B testing tools do very nicely (not in mvp)
  2. Javascript tracker (client side)
    • the Javascript tracker fetches the tests variants for this the web page or app,
      • (we need to let user in JS know about which A/B tests are running on the site. let's think of how we can avoid leaking all details of running A/B tests in the process?),
    • decides how to assign a visitor to a given test and given variant based on the variant probability,
      • maybe we start with a simple algorithm eg. 0.5 probability of A and B variants
      • (we may need to process the probabilities server side in case some of the variants will need to be given more or less traffic)
    • selected test variant info is persisted in a new first party cookie,
      • a user will be always shown the same variant (using this cookie we know which variant was served before),
    • Maybe we could reuse some code or ideas from this project for JS API
  3. Tracking API (server side)
    • Store the tests and variants viewed by a visitor, for each visit and goal conversion
      • do we also need to store variants for each action / page/ event?
  4. Reporting:
    • process the KPIs for each A/B test such as: % change in Actions per visit, Time on site...
    • process Conversion rate of each A/B test variant for each Goal known in Piwik (eg. Goal "Download brochure" conversion rate increased 50% in test variant A")
    • create an overview report listing most important metrics and in information, and also let user view all % changes in KPIs for each running A/B tests,
    • Let user know when test is finished, eg. this website calculator http://www.experimentcalculator.com/#lift=20&conversion=5&visits=1000

Other notes & ideas about A/B plugin:

  • when disableCookies is used, maybe do not disable the a/b testing cookie
  • let user know when a A/B test has enough data to be statistically significant.
  • often (for low-med traffic websites) it may take several weeks to get statistically significant results, maybe we could notify users that a A/B test(s) is finished?
  • maybe we could automatically pick the winning variant when a test is marked as "statistically significant better" (useful if user cannot directly implement the changes in site or app) )
  • A/B tests could be available as Segment so users can see all Piwik reports segmented by experiments
    • this also would let users open in one click, a segment of visitors that were running in variant [abTestVariantName] in [abTestName]
  • maybe it could work for tests created with other A/B tools that offer Javascript APIs to get the tests and variants (such as optimizely).
  • maybe in Visitor log we could show which tests & tests variants were seen by a given user and/or given action.
  • maybe users already use other A/B tests tools where they create variant and just want to "import them in Piwik" to measure them... supporting this may be complicated)
  • random testing idea: we should run A/A experiments and check that the A/B testing engine does not declare one experiment as being "statistically significant" better!
  • (not mvp) a A/B test could be customised to be served only to a particular "high value" segment (based on a Custom Segment in Piwik) (optimizely does something like this)

Keep feedback coming!


@mattab commented on April 28th 2014 Member

Interesting also: http://bentilly.blogspot.co.nz/2012/10/ab-testing-scale-cheat-sheet.html

Most Winning A/B Test Results are Illusory On the importance of running the power calculation: What is sample size needed? Indicate how long will the test likely take before reaching sample size.

@nkuehn commented on April 30th 2014

Hi erveryone, great to see that there's a lot of interest. What do you think about splitting the ticket into two:

  1. one about the proposed feature that allows controlling the test run from inside piwik (I agree with Matt that there's relevant market pressure to have some kind of feature like this, no matter how big or small it is - I personally don't need it, but that's not relevant)
  2. one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually), which are required anyways, no matter which way you run the tests (this new piwik feature, homebrew, optimizely, ... you name it).

The "only" thing that both definitely need is a conceptual decision on how the A/B comparison is realized in the piwik core (e.g. visual segment comparison vs. a specialized feature with own database structures vs. ... ). Probably the hard part...

@williampatton: I personally think that custom session-scope variables are more suitable to tag the visitors into A/A2/B/etc than events (conversion is the typical success aspect and conversion is usually session-scope, which events aren't). Unfortunately their number is limited as far as I understand the Piwik API.

@mattab commented on June 5th 2014 Member

Interesting article: What you should know about the Optimizely vulnerability which discusses the privacy implications of A/B testing implementations, and how, we should aim to avoid 'leaking' A/B tests details in the javascript of A/B testing users.

@tassoman commented on June 19th 2014 Contributor

I agree that having a full AB Testing feature would be having too much inside Piwik.

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

@mattab commented on June 26th 2014 Member
@mattab commented on August 7th 2014 Member

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

current experiment picked by A/B testing logic will be stored in a cookie and in a custom variable in piwik visit and page rows, and we may also use it to segment the report data for each experiment.

edit: interesting template: http://www.bloggerchica.com/test-design-the-official-doc/

@Tobias-Conrad commented on December 2nd 2014

Hi what is the status? How could A/B Testing be used?

@nkuehn commented on December 2nd 2014

FYI: I have a new Job and therefore can' participate with ideas and testing based on real world data any more. The comparison dashboard looks very nice as far as the screeshot can tell. Hopefully you get the funding campaign completed!

@marclaporte commented on March 23rd 2015 Contributor

20 lines of code that will beat A/B testing every time

What’s Happening In The A/B Testing Market?

Seeking Edge, Websites Turn to Experiments ; Optimization technology is reshaping publishers’ decision-making process—and the Web itself.

"On 1 June 2012, Google announced that GWO as a separate product would be retired as of 1 August, and some of its functionality would be integrated into Google Analytics as Google Analytics Content Experiments. Google Website Optimizer allowed webmasters to test an alternative version of an entire page, known as A/B testing — or test multiple combinations of page elements such as headings, images, or body copy; known as Multivariate testing. This tool was part of Google Analytics, though accessed through a different user interface. As it was part of Google Analytics, it used the Google analytics scripts."

@Tobias-Conrad commented on March 23rd 2015

ticket opened 2008, added to major 2014,
may it is time to give a easy solution made by piwik.
like https://clicky.com/help/faq/features/split-tests/how

@Tobias-Conrad commented on June 27th 2015

@mattab just do it. why not finished?

@abishekrsrikaanth commented on July 6th 2015

This is a PHP based implementation of Facebook PlanOut. https://github.com/vimeo/ABLincoln

@mattab commented on February 9th 2016 Member

the scientist project looks also possibly interesting https://github.com/daylerees/scientist

@mayannath commented on March 12th 2016

since 2012.. no further insights into this thread? This is an interesting topic.
Dev's please provide further details on Split-Testing mechanism through Piwik.

@mattab commented on March 16th 2016 Member

Today Google re-launched their A/B testing tool as Google Optimize. Here are a couple screenshots.

Google Optimize Editor


Example of Google Optimize Report


Source: https://www.e-nor.com/blog/google-analytics/google-analytics-360-suite-from-measurement-to-optimization

@tassoman commented on March 16th 2016 Contributor

It's quite big amount of informations but in my opinion is missing of page loading time.
More, I would draw the "to be Best" graph in green and the other in red.

@JLuc commented on April 14th 2016

Is there a simple way we can testA/B with piwik now ?

@tsteur commented on October 27th 2016 Member

We have released a plugin that works on top of Piwik to run A/B Tests, Split Tests, Experiments see http://www.ab-tests.net/ . A full list of features is mentioned on the Marketplace at https://plugins.piwik.org/AbTesting . It lets you easily run experiments on websites, servers, apps and in campaigns and has lots of features to customize your experiment and to define your success and your expectations for an experiment to increases your sales, revenue, conversions, pageviews, and more.

More docs are available at: User Guides, FAQ, Developer docs

@williamtheaker commented on October 27th 2016

That plugin is proprietary, which is quite disappointing given Piwik's emphasis on the importance of software freedom.

Can you reopen this issue until there's a libre/open source A/B testing plugin for Piwik? It's still on the Piwik.org roadmap.

@pattonwebz commented on October 27th 2016

I 2nd the notion that this issue should remain open until there is an open source alternative available.

@mattab commented on October 27th 2016 Member

We hope that although not open source, many will enjoy the new A/B testing product.

Researching, building A/B testing, documenting, testing, took months of work. By purchasing a license, you get a fully working product, with free updates for duration of license, and you stay in full control of your data.

And on top of this, you help the Piwik core platform engineers to grow and keep innovate. That's because a % of all non-open plugin license sales earnings directly go to funding Piwik engineering. In particular what plugins sales fund are:

Next week we'll publish a blog post on the Piwik blog, with a bit more information.

ok to reopen the issue as suggested.

@pattonwebz commented on October 27th 2016

Glad this is reopened and hopeful to see some kind of a core implementation for split testing some time in the future.

Also +1 for the licencencing fees contributing to core development. I've suggested to a client that this plugin is worth considering for them and am excited to get the chance to trial it in production.

@mattab commented on October 27th 2016 Member

Thanks @pattonwebz for advocating the plugin to clients, we do appreciate your support a lot :+1:


@mayannath commented on November 1st 2016

I think the plugin idea is great! Hope to see development on this front
Thanks for re-opening.

On Thu, Oct 27, 2016 at 4:50 PM, Matthieu Aubry notifications@github.com

Thanks @pattonwebz https://github.com/pattonwebz for advocating the
plugin to clients, we do appreciate your support a lot 👍

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/piwik/piwik/issues/459#issuecomment-256801544, or mute
the thread

Mayan N.

Powered by GitHub Issue Mirror